Keywords

Introduction

Multimodality has emerged out of relative obscurity over the past two decades as a term that has purchase in language and literacy research, theory, and practice. This chapter is concerned with research methods and approaches to the study of multimodality in language and education. Origins, early developments, and current uses of multimodality are discussed. Research methods that examine multimodal meaning-making operate across varied contexts and use diverse heuristics and research tools.

Multimodality explains communication as a combination of modes of representation and expression within text designs (with the term text referring to communicative acts beyond but including print or writing). The various multiple modes of expression can be visual (e.g., drawing, painting, video), print (e.g., books, newspapers, environmental print), gestural (e.g., miming, pointing, acting out), dramatic (e.g., role-playing, improvisation, formal acting), and oral (e.g., informal talk, public speaking) modes (Kress 2000; Stein 1998). Kress often stands as a harbinger of multimodality with his theory and writings over the years, but this field can be traced to earlier theorists such as Halliday (with whom Kress studied).

Major research trends in response to early developments have emphasized design, digital and visual literacies, juxtapositions of home and school literacies, and interests in material culture, to name a few. Current explorations of embodiment, Deleuzian approaches, material culture, cultural geography, and gamification have resulted in innovative research methods. Challenges of multimodal applications to educational contexts that move beyond representation and that address earlier claims and issues of equity and social justice are described.

Early Developments: Text-Centric Semiotics → Social Semiotics

Although the term multimodality became more prominent at the turn of the twenty-first century, language and literacy educators and researchers have been interested in other modes, besides print, since the 1970s (e.g., Clay 1975) and the 1980s (e.g., Dyson 1982). In practice, the term multimodal is used often when talking about visual or design literacies, digital literacies, or nonlinguistic modes alongside other modes. Print texts can also be considered multimodal if one considers elements such as layout, and font, as well as tactile features of a particular text, for example. Multimodality is not new although the ways in which texts are viewed as multimodal and the ways that non-print texts are valued are shifting.

The roots of multimodality in education are most closely connected to early theorists in semiotics, starting with Saussure (1959) and Peirce (1977), linguists who explored how signs are used to convey meaning and then more fully developed by Halliday, who presented a social understanding of semiotics. Semiotics works on the principle that anything can be a sign, as long as one derives meaning from it. Theorists working within a semiotic frame view language as an idealized and abstract system (Peirce 1977; Saussure 1959). Peirce and Saussure, two key semioticians, both recognized that semiosis does not happen in isolation. Saussure developed a formalized approach to semiotics that described how signs have meaning relationships to each other. Peirce, on the other hand, believed that people use semiotic resources at hand to communicate. One of Peirce’s well-known phrases is “we think only in signs.” Along with Saussure, he discussed the signifier as the form and the signified as the concept one derives from the form. In multimodal parlance, the signifier is the material and mode and the signified is how meaning is made. Both theories are complex and this summary does not do them justice; nonetheless, on the whole, what both semioticians foreground in their work is an opening up of what text is or can be, and the germs of their theories grew into multimodality.

Like linguistics, semiotics has a technical, and, at times, rigid, grammar. Theorists like Halliday working within a semiotic framework felt constrained by the rigidity of a grammar and this is reflected in his work. In the 1970s, Halliday developed social semiotics and how language is socially negotiated and socially constructed. While moving the field of semiotics into the social, he wrote that meaning arises from social interaction and is made through a “dynamic process of sign making” (Halliday 1985). With this social turn, Halliday showed how the making of signs and sense making through texts is inherently human and individuals use and make meaning from materials and modes within contexts.

In his writings, Halliday discusses how individuals make choices from different modes of representation and expression based on the situation and the audience, and through this reasoning he developed metafunctions. Halliday refers to metafunctions as fundamental properties of all signs. There are three metafunctions that constitute texts: (1) an ideational metafunction that refers to the ideas or concepts that represent meanings in a text/sign; (2) an interpersonal metafunction that refers to the target audience, the intended view to which the sign speaks; and (3) a textual metafunction that refers to the text’s design, to the physical, material choices that constitute the face of the ideas presented to the text’s audience. Halliday’s language of description provided more granular ways of describing meaning-making and showed how pivotal social mediation and subjective choices are in sign-making. Studying with Halliday, Kress then moved the field of social semiotics into multimodality as an accepted term within educational theory and practice. Some years later, Kress continued to build on Halliday’s ideas when he talked about sign-making as a metaphor for the ways in which meanings are multiplied. By the mid-1990s, multimodality became more prominent within literacy education and Kress and Siegel stood out as key multimodal theorists.

Major Contributions: Social Semiotics → Multimodal Literacies, Context, and Identities

In the 1980s and 1990s, when multimodality was becoming more central in language and literacy, several conceptual strands were present in the field: the elaboration of multimodality by Kress and Hodge and by Kress and Van Leeuwen; Siegel’s description of transmediation, meaning-making across sign systems; the formation of the New London Group and the publication of the pedagogy of multiliteracies manifesto; and various responses to the New London Group and others that were manifested through further research and translated into educational practice. During this period, which represented a social turn in literacy and education research, context and identities became more relevant. In this section, major foci of multimodal research, as well as new approaches to researching multimodality, are discussed.

Kress: Nuancing Social Semiotics → Multimodality

In the late 1980s, Hodge and Kress (1988) elaborated on the intricacies of social semiotics as a more nuanced way of regarding communication. While accounting for how modes of communication are actually used, Hodge and Kress developed flexible terms and concepts for ways that people used different materials and invested parts of themselves in their text making. Across contexts and situations, individuals choose which modes to privilege. Kress and Hodge opened up ways for educational researchers to think about subjective and social mediation of content.

Kress built on these ideas when he talked about sign-making as a metaphor for the ways that meanings are multiplied in texts. Offering quite radical (for the time) conceptions of meaning-making, such as motivated signs, Kress maintained that when a child or meaning-maker more generally composes a text, the text design and content are driven by the interests and the motivations of the sign-maker. If the sign-maker draws four circles to represent a car, then that is how the sign-maker sees the text and, by extension, this text is a window into their subjectivities. Kress used terms such as affordances and constraints and underscored the importance of affect and synesthesia as fundamental to how people design and “read” texts.

Siegel: Transmediation as Movement Across Modes

Siegel (1995), also starting with Peirce and Saussure, and building on Halliday, focused early on the generative possibilities when moving across modes (i.e. from writing to drawing). She relied heavily on Peirce’s understanding of any sign use as the expansion of meaning and elucidated the organizational rules of different sign systems. One of her greatest contributions to the field has been her ability to explain and illustrate how meaning-makers, particularly children, move across two or more sign systems (e.g., from words to images and then to gesture) and, importantly, how meaning-makers invent relationships between modes which enriches their understandings). Siegel argued that children use these generative potentials more fully as they move more easily across modes in their early play, until they learn how to work within more valued modes (such as print). She connects the generative potential of transmediation, or cross-modal movement, with the turn, in educational arenas, toward inquiry rather than transmission models of formalized learning. By complicating and nuancing meaning-making in this way, she demonstrated how young children represent agency in their learning.

The New London Group: A Pedagogy of Multiliteracies

In 1994, The New London Group formed to build an agenda for transformation of literacy practice, key components of which were the need to change what counts as literacy and an acknowledgment of the multimodal nature of literacy practice. The pedagogy of multiliteracies was intended to (1) shift what is counted as literacy and (2) acknowledge the multimodality of literacy practices. Kress had the longest history of theorizing multimodality and design dating back to his early social semiotic work with Hodge (Hodge and Kress 1988) and then on his own as well as with Kress and Van Leeuwen (2006). Within this work, children’s multimodal texts and the meanings constructed through these texts challenged the dominance and authority of print above other modes. The New London Group (1996) pushed for use of their pedagogical manifesto to reframe and expand literacy – and the importance of multimodality as a primary idea – in both research and educational contexts. In the wake of this landmark event and the edited collection that resulted from it, the multiliteracies framework has been applied across varied contexts and learning spaces to research language and literacy practices and to broaden definitions of communicative competence.

Early Responses to Siegel, Kress, and the New London Group

In response to the early work of Kress, the New London Group, and Siegel, many others took up the value of multimodal perspectives as foci for their research and for pedagogical purposes. Much of Siegel’s theorizing derived from detailed observational work watching young learners make meaning. By 2000, other researchers began to operationalize such terms and concepts as transmodal work, the motivated sign, and affordances and constraints of modes across different contexts.

Stein: Multiple Modes and Equity Research

The New London Group argued from a Freirean perspective for the need to critique the mainstream or popular face of education and provide access to learning for more people, in more ways. Interest in non-print modes, as well as recognition of the multiple modes represented through many textual forms, often connected to an interest in equity and the potential for an expanded understanding of literacy to allow for successful participation by more learners. A communicative ensemble evokes how the elements of an orchestra work to produce a performed piece of music. Stein argued that the consideration of multiple modes, especially those that are not based in language such as drawing and gesture, renders visible ideas, feelings, and meanings not as easily conveyed directly through language. Stein moved the language of multimodality into social justice work by observing how young children in South Africa moved across contexts from home to school to out in the community and built on their own awareness and backgrounds to make signs and how these untapped, sophisticated understandings can be built upon in school (Stein 1998). Other researchers describe the multimodality of communicative texts in terms such as bricolage, assemblage, and composition (e.g., Wohlwend 2009). All argued that when multimodal modes of expression come together, they create something more than the sum of their parts.

Ethnographic Research on Everyday Literacies

Researchers also began to investigate more fully how multimodality might be conceived outside of formal educational settings, across home and school contexts. Researchers like Pahl (2007) spent extended periods of time in homes, out in the community, and in schools observing, writing field notes, and, perhaps most importantly, collecting artifacts that they would analyze and lift out the choices made during production and the larger ideologies, values, and beliefs that these artifacts signal.

Although Barton and Hamilton’s (1998) 3-year ethnography of a town in the northwest of England may not identify as multimodal, their research methods drew on social semiotics and multimodal meaning-making. Local Literacies drew from a large corpus of data that were photographic, written texts, diagrams, signage, and their interpretative framework in the book drew significantly on the visual communication of everyday texts. Focusing on the social practices of everyday life took an anthropological gaze on the texts of everyday life. They sorted photographs into categories of events and practices. Using an artifactual, descriptive approach, they validated a diversity of everyday literacy practices.

Researching Up Close Across Time and Space

In many multimodal research endeavors, there were attempts to look at data in a granular way, and critical discourse analysis was often used. Using one form of critical discourse analysis , Scollon and Scollon (2003) applied the principles of social semiotics to their research method. They researched semiotics within particular spaces and how individuals move through semiotic systems while constantly reading and interpreting texts across spatial landscapes, albeit in a naturalized, tacit way. Similarly, Lemke (2000) supplemented multimodal theory by theorizing of literacy practices through timescales. Lemke explained that objects or artifacts carry with them longer and shorter timescales. An everyday object, like a household knife, does not carry the longer timescale and power that a samurai sword carries because the samurai sword has a much longer history and heritage. As Lemke described, “the material characteristics of the object also function as signs for an interpreting system of meanings that belong to processes on a very different timescale than that of the event in which the interpreting process is taking place” (p. 281, emphasis in original). Lemke’s timescales offered researchers a method to analyze the significance of everyday objects within an individual’s learning trajectory.

New Research Methods: Borrowing Across Fields

Research in the area of multimodality has been largely qualitative, often ethnographic, and engaged in close-up study of phenomenon. Certainly ethnographic and qualitative methods have often included the collection of visual or material data in the forms of artifacts such as drawings, photographs, and objects. The primacy of data collected in visual, auditory, or video forms has been highlighted through multimodal research. Regardless of the breadth of qualitative research, especially that following an anthropological tradition, the transcripts of speech from observation or from interviews have been the most cited and analyzed mode of data.

Works in Progress: Transdisciplinary Moves

At present, the study of multimodality requires the nimble use of conventional research tools and methods but also engages some in a quest for new tools and methods of analysis. Varied approaches to multimodality have evolved out of efforts to research across contexts and combine a multimodal interpretative framework with other theoretical perspectives, fields, and disciplines. An example of this approach is exemplified in a collection by Pahl and Rowsell (2006), featuring international research that combined multimodality with New Literacy Studies and anthropological perspectives to illustrate complexities at play during meaning-making and a nuanced blending of theoretical traditions. Below are more examples of these emerging approaches.

Co-curation

Coproduction of research and analysis alongside research participants has become prominent in the early twenty-first century, alongside an increased interest in the roles of art galleries and museum in multimodal learning experiences. Multimodal, environmental installations provoke learners (e.g., Hackett 2014), and those who build upon “children’s voice” research, originating primarily in the UK, have engaged in partnerships with participants and community members in formal and informal learning spaces (e.g., Jones 2014).

Emotions and Embodiment

More recently, there has been a push with for research that emphasizes the embodied and affective nature of multimodal meaning-making (e.g., Leander and Boldt 2013). Lewis and Tierney (2013) focused on how emotive interactions in race-related discussions were mediated by texts and visuals in an ethnically diverse urban school. They analyze how emotion is often separated from the mind and excluded from disciplines like semiotics, linguistics, and multimodal research, perhaps because emotion is laden with felt sensibilities and intangible aspects as feelings, beliefs, and embodiment.

Aesthetic Perspectives

Many semiotic and linguistic accounts of aesthetics have focused on the arts and artistic appreciation. Multimodal theorists have incorporated theories of aesthetics into their multimodal research in order to soften design grammar and to reflect on alternative perspectives on how aesthetic features signal agency.

Material Cultural Perspectives

Researchers of material culture have been preoccupied with how we interact with objects in our environments and how children engage in playful literacy work with toys (Wohlwend 2009). This multimodal literacy research values children’s contributions and expertise in their own explorations, and reflexive, sometimes parodic, multimodal play (Collier 2015). Brandt and Clinton (2002), using actor-network theory (Latour 2005), traced the movement of and relations to nonhuman actors (i.e., a book) in networks, looked at local-global connections, and argued that a “transcontextual” (p. 343) understanding of local literacies may not adequately account for all of the literacy ideas and objects with which we engage.

Translingual Perspectives

Globalization and its impact on communication and multimodality have been introduced by theorists like Canagarajah (2013) who use concepts such as translingual practice to highlight the point that traditional terms such as bilingual literacy or multilingual literacy create boundaries around language proficiency and cultural practices. Individuals usually draw on a range of genres, registers, dialects, and styles that render the concept of monolingual outdated. Researchers working across language education and multimodality have used multimodality to defy the mono-nature of language practices and to illustrate how multiple and multimodal language and literacy practices are (e.g., Kenner 2004).

Immersive and Virtual World Perspectives

Multimodality has lent itself to research in digital worlds (Marsh 2005) because digital worlds are fundamentally multimodal. Gee (2006) has been pivotal in highlighting the strength of video games in fostering new literacy practices. Indeed, research on learning in virtual worlds draws significantly on the role of different modes in guiding thinking and problem-solving practices. Positioning the game controls as nonhuman participants, DeCastell et al. (2008) have critiqued assumptions about rule following and gendered assumptions of video game play. They used a form of microanalysis of video segments that slow down the passage of time and embodied participation. Within online and immersive spaces, gendered, linguistic, and cultural identities can be modified or reconfigured with different impacts and effects that involve the coming together of modes.

Mobility Perspectives

Coupling theories of mobility and actor-network theory (Latour 2005) with multimodality allows researchers to build in spatial theory (Leander and Boldt 2013) into their interpretations of modal learning across different contexts. Indeed, the concept of semiotic mobility resonates with Deleuze and Guattari’s (1987) challenge to static and logocentric forms of thinking and meaning-making through their concept of the “rhizome.” Their delineation of the rhizome and of rhizomic movement allows for thinking about movement from one mode to another or transmodal work. This approach complicates multimodality because of the multiple and unpredictable ways in which individuals produce and read texts across sites.

Problems and Difficulties

Conflation of Design Literacies, Digital Literacies, and Visual Literacies with Multimodality

The primary challenge of multimodal approaches and research has been to tease apart design and multimodality, which have been used in almost synonymous ways. Reading Images by Kress and Van Leeuwen (2006) has contributed greatly to the reception and interpretation of and to the focus of researchers and educators on design. The term “design” was integral to the pedagogies of multiliteracies manifesto put forth by the New London Group who were interested in the digital as a place of expanding conceptualizations of literacy. However, digital and design literacies (Sheridan and Rowsell 2010) have sometimes been coopted in mechanistic or superficial. Design is only one part of the multiliteracies pedagogy and has been a focus because it naturally coexists with the desire and need to understand, interpret, and value the visual, in formal classroom settings and in the online world. Design literacies that follow Kress and Van Leeuwen’s grammar have a Western focus and have not always been helpful for visuals produced in non-Western settings.

Retaining Complexity of Multimodally: How to Capture Multimodal Data

Another challenge for multimodal research is how to retain the complexity of multimodality and both the ways in which modes come together and the affordances of modes singly. Bazalgette and Buckingham (2013) argue that multimodality has been used to dichotomize print and non-print texts. They also argue that multimodality’s inherent complexity has been oversimplified in its translation from research to practice and that the importance of interaction between modes is not captured by current research and practice. The ways in which multimodal data is generated, captured, and analyzed in ways that represent its multimodal nature are still a challenge. For example, with multifaceted, multimodal approaches, researchers are challenged to situate physical movement of an individual alongside swiftly changed web-based images. Of particular concern is how one can analyze data that is collected in asynchronous or nonlinear ways, and then, further, how to present multimodal data when reporting findings, where even illustrations or other non-print representations are still resisted in established publications.

Spaces Between Modes and Nonlinear Representation

Connected to concerns with process and the “how” of multimodality text making, Siegel and Panofsky (2009) argued that what is absent from semiotic consideration of multimodality is an “understanding of what people make of the space between multiple modes” (p. 101). In response to this concern, they turned to sociocultural theory to provide meanings beyond multimodal analysis (or an examination of “site of appearance,” “display,” “narrative,” and “genre”) that might account for texts’ origins as well as “what human beings do as organized in activities that are practiced by social groups” (p. 105). Although one may often consider texts as finished products, such as a published novel, a publicly hung painting, or a choreographed dance work performed for an audience, here texts-in-process or drafts are considered as worthy of examination for the influences that are rendered invisible when one looks only at final products. Following Deleuze and Guattari’s (1987) rhizomatic, nonrepresentational theory of how subjectivities and literacy moments are produced in nonlinear ways, Leander and Boldt (2013) have argued for an understanding of meaning-making that is grounded in diffuse and embodied experiences of the world. This unbounded nature of communicative practice can include multilingual and digital forms, often unrecognized within traditional governance structures, but recognized within contemporary society and local spaces. The influence of cultural geographers (e.g., Massey 2005) is seen in the ways that literacies are presented in ways that stretching across the fluid and changing elements of time and space.

Challenges of Application

In many areas of multimodal literacy and multiliteracies research, the hopes for equitable and transformed educational practice have not been fully realized. Critiques of multimodal work can be, at the same time, critiques of schooling (Jacobs 2013/2014), and the entanglement of schooling and multimodality continues. The New London Group’s manifesto and its shortcomings in practice constitute a call to revisit and highlight the commitment to equity originally expressed and the potential for multimodal literacies to address these issues.

Future Directions

Multimodality has widespread hold as a way of both understanding and expanding literacies, how meaning is communicated and created through ways of interacting in contemporary times. As with all fields of research and inquiry, the landscape of multimodality is shifting. Based on its history and present directions, it is likely that certain areas will come into greater focus and clarity. Based on this audit of the field to date, the following pathways are perceived as future foci for multimodal research: a return to the New London Group (1996) manifesto to build on these original ideas – the “twin goals of access and critical engagement” (p. 96); more work on social justice and equity studies across international contexts; more transdisciplinary research drilling deeply into language and linguistic and cultural diversity; an emphasis on big data and multimodality on Twitter and other large repositories of quantitative, visual, modally complex data; and research on wearable computers and gaming heuristics and epistemologies (e.g., Minecraft).

Future understandings of multimodality need to continue to be grounded in both offline and online worlds (without dichotomizing these), should consider the affordance of modalities (e.g., visual versus auditory modes), and also are called to explore in finer ways the complexity of modes that come together in multimodal literacy moments, events, and representations. The identities (or subjectivities) of multimodally literate beings are grounded in a wide range of overlapping and changing social investments and are the result of embodied, sensory, and diverse affects and effects.

One example of a generative vein of inquiry for future multimodal research is exemplified by Burnett’s (2015) argument that today children, adolescents, and teenagers move in and out of physical spaces and nonphysical, virtual spaces that shape their understandings about the worlds. In this work, she contends that literacies which relate to physically present objects and texts coexist and seep into more immaterial literacies which are materially absent and intangible but nevertheless integral to meaning-making such as memories, feelings, and even virtual worlds. Her approach decenters mechanistic work on digital environments as a panacea or as a tool, and through data from her research and other research, Burnett maintains how complicated and rhizomatic moving between online and offline worlds can be.

When working with multimodality, and a multiliteracies perspective, these tensions are inevitable and can be productive. Additionally, a view of multimodality that keeps critical perspectives, equity, and social justice at the forefront is one that can potentially lead to educational change and considers how multimodal perspectives allow for one to see what is happening differently and for one to recognize and value the potentialities of various modes and modal compositions.

Cross-References

Related Articles in the Encyclopedia of Language and Education