There is some degree of convergence in the theoretical, research, and practical lenses applied by each person involved in this special issue—with distinct differences in their perspectives and underlying foundations and views about the roles of the form-function of language in doing and learning science. However, there is common agreement on the importance of multiple representations, multiple modality, and science literacy for all. A necessary and critical element of this emerging area of research is to engage the central ideas or concepts that frame the work on understanding the role language plays in constructing and communicating science for learners, whether they are scientists, teachers, or students. Here the concept of language is used in its broadest sense; that is, language involves all forms of representations that are used within science, and the most effective scientific communications are multimodal. Therefore, science literacy partially involves competency with the various modes of language.

School science tends to emphasize verbal and mathematical language, but the changing technological context of information requires consideration of other forms of representation (Gilbert 2004, 2008; Trumbo 1999, 2005). A cautionary note here is to avoid thinking of language proficiency across the modes as simply visual literacy and that all representations are models. The intersection of these elements for any learner can be framed by the work of diSessa (2004), who suggested the need to talk about a learner’s multimodal representational competency (MRC). He argued that MRC is more than just the “mere production and use of representations [but rather] stands as a free resource for further learning” (p. 294). By engaging their MRC, students are able to build knowledge rather than simply recall and regurgitate the sanctioned representations supplied by textbooks. If we are to engage learners in the languages of science such that they are able to have authentic experiences and grasp the material and social practices of the scientific communities and scientists (Ford 2008; Ford and Forman 2006), then we need to examine what competency means.

While we know very few students go on to be scientists, the important issue for us is how to help all learners engage in disciplinary practices that allow them to construct rich understandings of science, feel comfortable with the practices, traditions, and conventions of scientists, and participate more fully in the public debate about critical science, technology, society, and environment (STSE) issues—signs of science literacy for all. Tang and Moje (this issue) emphasize the need to provide theoretical foundations for understanding and applying multimodal representations in discipline-specific literacy and the progression from generic literacies to disciplinary literacies. They unpack of the triadic semiotic model used by Hubber, et al. (this issue) and Waldrip, et al. (this issue) to illustrate the difference between making meaning with representations and making meaning from representations. Klein and Kirkpatrick (this issue), like Tang and Moje, focused on the transformations and movements amongst the three nodes of the semiotic model, and the active transforming of knowledge between representations, or resemiotization, in the meaning making process. They stress the supplemental and special power of “resemiotization” to scientific inquiry as learners make measurements, image or construct visualization, and build mathematical or canonical representations in science. Clearly, MRC and science literacy for all involve and require both theoretical (cognitive action) and pedagogical (learning and teaching use) considerations for future research, curriculum development work, professional development projects, and implementation efforts.

Unpacking Fundamental Literacy in this Special Issue

Positioning multiple representations, multiple modality, and textual, semiotic and symbolic modes within the larger framework of science literacy for all, can be seen by realizing that these modes and their use in science play roles in both the understanding of science and the fundamental literacy in science that allow scientists and students to construct understandings and to report and argue these ideas with others. Formulating and using models, a single form of representation, are valued, integrated processes of science; they are subcomponents of MRC in science; and the resulting models are essential ideas in the understanding of science. Central themes across the studies in this special issue are the implicit cognitive symbiosis of multimodal representations and the moving amongst these representations on science understanding (theory of action) and the explicit pedagogical effectiveness of student-generated representations in science learning (theory of use). Further research and development activities related to MRC would benefit from serious consideration of theory development based on classroom use and on cognitive sciences. Searching for the silver bullet with a learning styles framework does not appear to be a strong approach or theoretical foundation since it is not aligned with the material and social aspects of science (Ford 2008) and with contemporary models of learning, especially learning as interactive-constructive negotiation within a sociocultural context. Some research and development programs appear to be focused on providing the right representation of the target concept to specific learners; several computer-assisted programs appear to emphasize such silver bullet foundations. Other research has exclusively focused on productive pedagogical moves that use representations to reduce complexity, provide analogical links, and make abstract ideas more concrete. We subscribe to a reworded proverb: Give a learner a representation and he/she may learn the concept, but encourage a learner to develop insights into creating representations of mental images/ideas and she/he will learn forever.

Theories of Cognitive Action

Building on the concept of encouraging students to develop insights into creating and using representations to construct understandings of science means that we must take seriously the concept of learning. Klein’s (2006) argument—about first generation cognitive psychology vs. second generation cognitive perspectives that questions the assumptions about expository genres and reassesses the role of narrative genres in learning—allows us to begin to pose questions about possible cognitive actions engaged by students and teachers when developing MRC. Most research and theoretical considerations on representation have focused on making meaning of prepared text as reading comprehension. Much of this work is based on Paivio’s (1978) dual-code model, which posits that words and visual information are processed on parallel interlocking pathways. More recently, several researchers have modified or revised parts of Paivio’s model. Mayer (2009) proposed a cognitive theory of learning from multimedia that is derived from theories of dual coding, cognitive load, and generative learning. He posited that meaningful learning involves five cognitive processes: word selection, image selection, word organization, image organization, and the integration of words and images. The cognitive theory predicts that both verbal and visual working memory are utilized during learning, working in parallel to produce two types of mental representations that are then integrated with one another (Mayer 2005).

Schnotz (2002) proposed an integrative model of words and picture comprehension that emphasized mental representations of multimodal texts and predicted the involvement of visual and verbal processing systems resulting in a more elaborate process than implied by a secondary coding of information. Because there are interactions between words and pictures, there is not a one-to-one correspondence between internal and external representations, that is, both words and visuals can lead to either descriptive (verbal) or depictive (pictorial) mental representations. Lowe (2003) and Rouet et al. (2008) provided a model for dynamic visuals, oral statements, and print that reflects the reality of ICT and multimedia information sources. Importantly, this work has focused on the development of a first generation cognitive perspective in that knowledge is about denotative forms of science. Hence, the efforts are about the construction of representations that will enable the learner to understand the concept. A question arising from such a position is: Whose knowledge is being engaged: the teacher, the student, the textbook manufacturer, or the scientist? These theories have been useful, but they have not provided robust predictions and powerful explanatory mechanisms for our current work on learner-constructed representations and the transformation amongst modalities. The central issue now is: How does this help us understand how scientists and others come to construct representations, and how do they use them to generate new knowledge and to communicate knowledge?

A second-generation cognitive position may provide a richer alignment between the cognitive actions needed to build MRC and current views of learning approaches being advocated for the learning sciences (NRC 2000, 2007). Klein’s (2006) argument for using writing-to-learn approaches that encourage more narrative forms of text as a means to help learners clarify their fuzzy understandings of science concepts may be a useful place to help develop richer explanations of cognitive actions required in developing MRC. The use of writing narratives to help clarify early conceptions and understandings is based on the idea that text construction enables the learner to use knowledge constituting processes (Galbraith 1999), which is a fundamental assumption embedded in our working definition of science literacy. Building on from this position, a number of questions arise:

  • How does constructing different representations of a concept help students clarify their understandings of a concept?

  • What knowledge bases are needed?

  • How do these knowledge bases interact?

  • Are representations developed separately or is there an interactive iterative process between different knowledge bases and multimodal representations?

These questions are not about what textbook manufacturers and authors think is best or what the teacher wants to use. Rather, these questions are framed to address issues related to how learners in a science classroom negotiate and construct their knowledge claims and understandings. If learning is about negotiation within and amongst learners, sensory experiences, prior knowledge, and sociocultural context, what are the negotiations required to move between the arrays of modal forms used to represent a science concept? Do some progressions appear to be more authentic in terms of science and effectiveness for the students doing them?

Any theory of action that will move us forward needs to consider, integrate, predict, and explain learner-generated representations, the transformation across multiple modality representations, and the negotiations involved in making sense of constructed and prepared representations found in various information technologies and sources, not just science textbooks. Ramadas (2009) concluded from a selective review of related literature in cognitive science, developmental psychology, science literacy, and science studies, that language-based reasoning is inseparable from and complementary to visual forms of reasoning. He stated:

It is less surprising, from the viewpoint of science, that model-based reasoning turns out to be a combination of verbal and visual, as well as symbolic and mathematical modes. For science education then we need to find out how best semantic content can be integrated into visual representations, in order to bring about meaningful learning.(p. 303)

Developing rich explanations of the cognitive actions used during these negotiations would provide the science education community a much stronger knowledge base on which to develop appropriate pedagogies of use. The explanations would allow for a much better alignment between teaching and learning such that teaching would be in the service of learning (Hand 2007). This is critical given the emerging technological advances and new representational opportunities that arise. We need to better understand how students use these explanations to construct understanding, rather than seeing them as silver bullets that will somehow transform learning.

We believe that the transformation among multimodal representations has the greatest potential in promoting learning and depth of processing. The embeddedness of representations, experience, argument, and printed words appears to be an indicator of successful integration of mental images, conceptual understanding, and stored meaningful knowledge. Embeddedness is least likely if the visual adjunct to printed words is decorative, more likely if it is representational or organizational, and required if it is interpretational or transformational (Carney and Levin 2002; Tippett 2008). Collectively, the current theories about visual, symbolic, and print representations do not appear to be robust enough individually to explain or predict the learning and pedagogical effects starting to appear in the research studies described in this special issue (e.g., where learners use visualization, material resources, physical manipulation, plausible reasoning [abduction, induction, deduction], and technologies to create or construct their own representation of an idea or event). We think that integrating ideas about cognitive psychology, nature of science, and linguistics, dual codes, visualization, models and model formation, types of representation, and systemic functional linguistics (Fang 2005, 2006; Halliday 1975; O’Halloran 2004; Unsworth 2001) might be a productive foundation for theory building or model enhancement (Hand et al. 2003; Yore et al. 2004). We need revised models or theories regarding multiple representations that focus on transformations across multiple modes and learner-generated representations.

However, we provide some cautionary notes within this encouragement. We are not convinced about the value of using the label mental models for private and personal mental ideas and images (Gilbert 2004). Scientific models have specific requirements; not all representations will qualify as models while all models are representations. Ideas and visualizations in our minds are unlikely to meet the conditions required of scientific models. Like Ramadas (2009), we believe it would be counter-productive to assume that visualization is separate from oral and print-based language forms (sketching, gesturing, word associations, etc.) and transformations, “imagery has some special status over language” (p. 304), especially print-based language, and creating representations exclusively involves abduction. We fully accept that construction of some representations (mental, linguistic, visual, symbolic, physical) involves gestalts or metaphors in which the unified whole, configuration, pattern, or organized field occurs to the person and the spontaneous occurrence cannot be distilled into a series of analytical steps or parts. But most of the construction of representations reported in this special issue involved either inductive or deductive reasoning where people combine specific experiences and observations to form general, regularized patterns of characteristics or they speculated the existence of features from a generalized understanding of an event to construct a representation. Furthermore, we believe transformation across representational modalities (making measurements and collecting data, designing and organizing a data table, constructing a data display like a graph, labelled diagram or flow chart, and describing orally or in print the patterns in the data display) encourages deeper processing and better understanding of the embedded ideas. However, we are not convinced this involves transformational reasoning as anything different from plausible reasoning, critical thinking, and creative thinking established in the science education literature.

Theories of Use

The term theories of use refers to the ways that an umbrella-like, integrated framework can promote and encourage the use of multimodal representations in classrooms and thus help students develop MRC. The articles in this special issue clearly show that there are multiple pathways that can be used in classrooms to engage students in developing MRC; that is, there is no silver bullet representation or pedagogical strategy that ensures students develop their MRC. Building on the extensive writing-to-learn research by all the authors in this special issue, the development of appropriate pedagogical approaches would appear to have a foundation in text-based work previously reported. The authors’ earlier work has been based on the concept that student engagement in language-based practices needs to be embedded into the normal science classroom practices in which they generate reports of inquiries, persuasive and informative messages about science ideas, and compelling arguments to support knowledge claims, and not be viewed as a separate component of science activities. As such, students have been asked to write for different audiences, using a range of different writing types across a broad range of purposes. We believe that these basic conditions should still be adhered to when asking students to construct presentations or products that are multimodal in nature.

What does need to be considered is the much greater array of representational opportunities that are now available for students to use both in constructing representations and in communicating these representations. The range of technologies has meant that the concept of production of physical models of, for example, matter/molecules, has expanded beyond the simple ball and stick model of previous times. The development of ICT has meant that representations are now available in forms not previously used. For example, real-time data capture and display, data generation and graphing, and dynamic visualization of atomic level phenomena enable students to engage in using representations to build understanding of the concepts. Presentation software enables students to build complex media presentations and data displays to communicate and negotiate potential understanding of the concepts under study. However, the availability of such technology does not automatically mean that students will be competent in its use or potential use nor understand the connections amongst the information/data, representation, and concept (Patterson and Norwood 2004). For example, we are all familiar with students’ inability to estimate the magnitude of number and their belief that what appears on a calculator is the right answer or to calculate average values for nominal codes like gender, colours, postal codes, or telephone numbers.

Thus, we need to explore the range of pedagogical approaches that will help students build MRC. These approaches will have to engage with the emerging technologies, build from current and previous pedagogies that have dealt with language as a learning tool focus, and be willing to expand the possible range of representational opportunities provided to students. We would stress that the need for broadening our current thinking is not a call for the production of science fiction in terms of the potential products, but rather the promotion of different avenues for students to build strong scientific conceptual understanding across the various modes used to frame the concepts about real science.

Pedagogy needs to consider how to support students as they negotiate meaning and communicate their ideas to others and acquire canonical understandings of scientific representations for the target ideas (e.g., force, matter). In the Huber et al. (this issue) case studies, students were provided with progressive supports that encouraged them to assume the conceptual understanding of forces and conventional representation of vector quantities indicating magnitude and direction by the length and head of an arrow applied to an object where unbalanced force causes some action. The teaching sequence was exploratory, but it illustrates the union of content and conventions with the scientific discourse community. Waldrip, Prain, and Carolan illustrated another pedagogical opportunity and barrier when the plasticine models of solid matter were to be modified or enhanced to demonstrate kinetic molecular motion about a fixed point and the increased motion overcoming the intermolecular forces during change of state. The limitations of the technology (plasticine models and digital cameras) did not allow the students to fully achieve their good intentions and accurate insights. Here is an opportunity for teacher scaffolding further negotiations and explorations by using technology or modes of the digital camera that would allow dynamic systems—video to capture the motion of the molecules. Another way would be to encourage students to take a series of still photographs and produce a time-series or time-lapse collection or a flipbook to illustrate relative motion of the molecules. This chance situation provides a just-in-time opportunity for direct instruction about relative motion, motion pictures, flipbooks, and video-mode on the digital camera. These skill instructions embedded into authentic negotiations illustrate the insertion of instruction when needed—MRC cognitive apprenticeship.

Closing Remarks

We believe that, like historical examples where engineering moves ahead of science, the applications (uses) of multiple representation and multimodal representation in science education are ahead of the theoretical foundations for these constructs and cognitive actions. Visualization and spatial orientation applications in chemistry and mathematics have been possible with technological advances in research laboratories and are required by the abstract nature of the target concepts and operations being investigated. But most of this work has occurred without the aid of a theory or theories to predict and explain the outcomes. Similar advances have been made in science education classrooms in elementary, middle, and secondary schools using much lower-level technologies.

The consideration of prepared multimodal texts in which readers make sense of the semiotics, semantics, and types of visuals, sign systems, and writing have been well served by existing theories and linguistics frames. However, they have not been as informative during our considerations of learner-constructed representations and multimodal texts. These various modalities are constructed in a time-series transformation in which written text is transformed, translated, or enhanced with mathematical formulas, scientific symbols, discipline-specific terminologies, or visual adjuncts or in which graphic or physical representations are transformed into a different modality or re-interpreted and elaborated with oral or written language forms. Hand and Choi’s study of meaning-making and embeddedness of multimodal text in organic chemistry revealed interesting potential to document the cognitive and metacognitive processes, language forms or modalities, and the resulting arguments and knowledge claims using a mixture of advanced technologies and writing. Their work extends the visuospatial thinking, visualization, modeling, and spatial orientation research in chemistry education into the argumentation and writing-to-learn science education research. Our post hoc consideration using the existing theories of multimodal actions and their experiences has not provided the insights required to fully predict the outcomes or explain the results.

Drawing, guided imagery, graphic organizers, mapping, and other visual representations have long been successfully used in early childhood settings to formulate, store, and communicate ideas. We have reported on exploratory inquiries into transformations among representation forms and found that some progressions appear to be more effective than others (Hand et al., 2007). We also have found that a number of clever ideas are used and recommended without sufficient evidence or theoretical foundation for such actions and claims. We are concerned that ICT and the advocation of learning styles will result in the uncritical promotion of silver bullets that will not deliver enhanced science literacy in its full vision of MRC and the symbiosis between and within fundamental scientific literacy and understanding. Collectively, the results reported in this special issue, our understanding of the research literature, and our experiences support the need for more comprehensive theories of action and use involving multimodal representations and texts.