Scenario

Two boys sit at a computer screen watching a video of themselves working on iPads using the app GarageBand to create a recording of an advertisement that incorporates voice-over with a musical theme. One researcher stands behind the students filming their interaction with the computer screen and each other. A second researcher interviews them about the learning processes they went through to turn their original notes about fast foods from a brainstormed text on butchers’ paper to a digital sound recording. The iPad video shows two boys sitting on a carpet working away from the rest of the class; their intent is to create an audio text that meets the criteria set by their teacher using different modes (such as print and sound in music and voice tone) to communicate a persuasive message. The video of the boys looking at themselves on the computer screen shows two boys explaining to two researchers a critique of their ability to collaboratively adopt the affordances of technology working with prescribed classroom limitations. In the shift between the private improvisation and the public reflection a semiotic reconfiguring of the GarageBand audio text occurs as a video recording, which enabled the students to re-examine their appreciation of how the use of the iPad app enriched the choices they had in creating their text.

Introduction

The introductory scenario is an episode from a 2-year empirical study run in a Grade 5 classroom of 11–12-year-old boys in an urban primary school in NSW Australia, where all students had their own iPads. The students were about to sit a standardized national literacy test that would require demonstration of such text features as correct grammar, spelling and effective vocabulary choice when writing a persuasive text. Rather than just teach to the test, the classroom teacher took advantage of the iPad affordances to integrate support for students’ development of linguistic knowledge with awareness of multimodal communication techniques. As the teacher had a master app through which student iPads could be controlled, work completed by students in isolation tucked away in a corner of the room could be harvested and displayed on the Interactive Whiteboard (IWB) at the swipe of a finger. Each time a text was repurposed in this way it became a public document which on one hand provided a stimulus for new learning to the whole group but also provided opportunities for individual students to reflect on their work. The uptake of iPads/tablets in schools worldwide has increased year-on-year but as yet there has been little research into the modal complexity offered by this learning device that takes into account the context in which learning takes place . While there is no doubt that mobile tablet technology brings access to new affordances , both the material and social nature of learning and literacy with these platforms needs problematizing in order to investigate implications for pedagogy .

The research discussed in this study defines an emerging theoretical concept—multimodal layering —as a way of investigating the complexity of cohesive meaning making when semiotic systems interact in a text which is then recontextualized in new semiotic configurations. The chapter foregrounds this concept to demonstrate the impact of socially situated and technologically mediated experiences of literacy on students’ learning with iPads in the primary classroom. Although the one to one use of tablet technology suggests that learning might become highly individualized when using these devices, this study shows that the use of the touch pads provided collaborative opportunities for both public and private learning.

The research took a theoretical stance built on multimodal/semiotic theory informed by studies of twenty-first century learning and literacies. A methodological perspective was adopted that viewed videos, student ‘think alouds’, student print and digital work samples as data. Using qualitative analysis to interrogate modes or combinations of modes (spoken and written language, image, sound, movement, gesture) the study provides insight into students’ learning with iPads. Our deliberations have led us to further explore the related issues of dynamic materiality and meta-awareness. We have defined dynamic materiality elsewhere as ‘the way touch technology enables the constant shift between modes and texts through which students need to navigate to build cohesive layers of meaning’ (Walsh and Simpson 2014: 102). The iPad platform is particularly generative in this regard. This chapter explores what multimodal layering is in relation to tablet use and how such layering is significant in providing rich learning opportunities for students. It concludes by arguing that the metaphor of multimodal layering helps to identify dynamic instances of semiotic and cognitive complexity.

Theoretical Perspective

The study is informed by complementary theoretical frameworks belonging to the semiotics of multimodality , (Kress and van Leeuwen 2001; Jewitt 2009; Kress 2010), and twenty-first century learning, (Wohlwend 2010; Groves 2012; Rowsell et al. 2013). The theory of multimodality has enabled researchers to explain and explore the way meaning is made both separately and simultaneously through modes of communication such as language, image, sound and gesture. It has been particularly significant in examining multiple literacy practices within digital frameworks. More recently, we have investigated the significance of touch as a mode for meaning making with use of tablets/iPads (Walsh and Simpson 2013, 2014; Simpson and Walsh 2014) and we have shown evidence of the dynamic learning processes that can occur in primary school students’ interactions with multiple modes on screen while interacting with their teacher and peers in classroom tasks. Through our research, we have investigated the nature of the literacy demands made on students when multiple screens and modes are accessed through tablet use. Our findings have led us to consider the impact of multimodality with the use of digital technologies specifically.

For example, in examining the complex processes that can occur with the use of digital technologies, we have considered other metaphors that address the way meaning can be articulated between and across semiotic modes along with the nature of ‘materiality/immateriality ’ (Leonardi et al. 2012; Burnett et al. 2014) of gesture and touch as we developed the concept of ‘multimodal layering ’. To explore this concept epistemologically, we now examine the different metaphors used to clarify how our theoretical perspective of multimodal layering contributes to the field.

Epistemological Metaphors

To investigate how researchers (including ourselves) represent concepts of multimodality , we first turn our attention to the implied connotations of terminology used in their research. Theorists have borrowed metaphors from a range of disciplines such as mathematics and science in order to make concrete their epistemology. Each one of these metaphors construes meaning with particular semiotic impact positioning the reader to comprehend abstraction through mental associations. We present a number of epistemological metaphors below noting how each one provides a particular perspective on the scenario shown earlier.

The term transduction , used in the fields of biology and physics, is used to describe a change in state in a one-to-one relationship of something recast into another form, whether it be genetic material or electrical energy, for example. In explaining the semiotics of multimodality, Kress states that transduction is the ‘…process of moving meaning-material from one mode to another’ (Kress 2010: 125) so that meaning is represented in a different mode, as in a verbal narrative being represented visually. As Kress comments this is a common process that occurs in many different types of communication. So transduction is useful in understanding the re-articulation of meaning from one mode to another but it does not convey the complexity of meaning that can occur with the convergence of multiple modes. Similarly, Iedema’s (2003) use of the term ‘resemiotisation’ describes the way meanings are transformed from one mode to another in different social contexts. This epistemological metaphor would only account for the part of the scenario where written text was realized as image such as when one student drew a cartoon on their butchers’ paper notes to represent the overweight person described in statistics in the fast food fact sheet.

As another example, the term multiplicative references mathematical conceptualizations—when more than one mode occurs in a communicative event then the impact of the modes is increased in a compound effect. That is, in this metaphor the semiotic impact of individual modes does not simply add up, one to another. Rather the modes interact with each other so that the meaning is complicated through combined semiotic impact. Lemke exemplifies the term as ‘the meaning resource capacity, of multimodal constructs is the logical product, in a multiplicative sense, of the capacities of the constituent semiotic resource systems’ (Lemke 2002: 303). With this metaphor, the theorist provides an image of expansion, which binds elements together in relation to each other as meaning making boundaries shift. This epistemological metaphor would only account for parts of the scenario when video image and audio interact to create a complex text demanding multiple forms of decoding. For example, when the twin audio tracks from the video within the video need to be understood as relating to different recordings of the boys working on their iPads .

By contrast the term modal density references scientific conceptualisations—when more than one mode occurs in a communicative event then the impact of each mode increases relative to the others. That is, in this metaphor the semiotic impact of individual modes does not merely become more intense. Rather the modes interact with each other so that as a whole they take on more semiotic weight. Norris exemplifies the term as: ‘Modal density refers to the intricate interplay of various modes of communication or the intensity of a certain mode that a social actor employs’ (Norris 2004: 102). This is a metaphor that contracts attention inward to collect elements together one in relation to each other. This epistemological metaphor only draws attention to the scenario text as a bounded whole privileging perhaps image as the mode with most potent impact.

So, in order to take account of what we see as semiotic reconfiguring that is prompted as students shift constantly from private to public learning spaces , we need a term that allows for that complexity. While the above three terms provide insight into the characteristics and quality of semiotic interactions as texts we offer the concept of ‘multimodal layering ’ to further explore these interactions taking into account the contexts in which they play out. This metaphor attempts to shift the focus of analysis to how semiotic interactions contribute to potential points of meaning making coherence. For example, in relation to the scenario at the beginning of this chapter, this epistemological metaphor accounts for how the audio of the GarageBand video is repurposed and becomes part of a new text as the researchers’ prompt for the student’s think aloud. That is, the new text reworked the modal configuration that the students had experienced to serve new purposes yet carried fossilized within its multimodal layering its own semiotic history.

We chose the term layering to reference geological conceptualisations, which encourage perception of landscape. We have selected this discursive viewpoint to take into account not just the isolated semiotic systems evoked through modes but also the ontological impact of the social context within which a text is interpolated and the meaning making events take place. For example, the boys’ original handwritten notes on the disadvantages of fast food are used during the learning sequence to create a digital text combining spoken words and an audio track, which becomes an oral presentation to the class. There is simultaneously a transference of meaning across different modes as well as accretion of meaning from the start of the lesson to the end. To deal with this complex situation the boys need to deal with multiple forms of decoding and encoding to incorporate, understand and respond to the language of persuasion including appropriate speech tones and sound effects.

In proposing this concept, we acknowledge that various theorizations of modal complexity already exist. However, we see the need to expand these conceptualizations by investigating not just how modes interact at the textual level but also how modes interact at the contextual level, in this case the classroom, which leads to changing meaning making possibilities. This metaphor thus enables the researcher to attempt a semiotic analysis of the multimodal literacy demands within learning events that incorporate literacy in digital communication as individuals interact with text in private or public spaces (Walsh and Simpson 2014).

We have suggested that the multiplicative effects existing within the semiotic boundaries of a text alter when multimodal layering occurs as the text is repurposed in a new interaction and the learner is repositioned to respond to the reconfiguring of semiosis. For example, because of the interview context where the video data of the boys’ work became a prompt for reflection, the students had to deal with an additional layering of modes as they discussed their learning across the different stages of the classroom tasks. Our data show that students’ learning is enhanced when they have to deal with multiplicative effects experienced in new contexts.

We propose that what we traced during the classroom interactions using digital texts of various kinds reveals evidence of students creating complex meaningful connections as a result of enacting different literacy practices through sedimented layers of semiotic modes. This conceptualization of learning is similar to what Tierney et al. (2006) described as a ‘complex layering of concepts’ but it is also different as it takes account of the fluid and immaterial nature of meaning making prompted in reading and writing with iPads as well as its physicality. That is, rather than focusing merely on static texts and concepts we have incorporated data, which records social interaction in dynamic learning events. In this way, we can attend to the importance of attending to the ‘here and now’ including what Leander and Boldt (2013: 24) refer to as the ‘sensations and movements of the body in the moment-by-moment unfolding or emergence of activity’ as well as the pedagogy that enables the action. The semiotic history accumulating in the layers of learning summarized in our scenario, is illustrated in Fig. 5.1.

Fig. 5.1
figure 1

An example of multimodal layering

Full size image

Figure 5.1 represents how the epistemological metaphor of multimodal layering accounts for the semiotic reconfiguring that occurred in the specific classroom experience summarized in the opening scenario. It indicates how reading (about fast food facts) informed writing (of a persuasive text) which was recorded with music, broadcast and reviewed. We explicate the process further by providing descriptive analysis (See Exemplars 1, 2, 3, 4, 5 and Coda) and extracting key themes that emerge to be explored in the discussion.

Methodology

Conceptualized as a small-scale qualitative, interpretive case study, the research is informed by qualitative methodology due to its focus on interaction in the social context of the classroom when iPad/tablets were used by primary school students in English/ Language Arts lessons. This kind of study is best explored in a case study, which allows the collection of rich data garnered from a variety of sources and through close observations over a period of time (Yin 2012) that captures the ‘local specificity’ (Dyson and Genishi 2005: 3) of a particular classroom. The study does not pretend to have high generalizability but can assert that it achieved useful findings, which will be relevant to teachers and teacher educators in showing the potential of multimodality to provide supportive scaffolds for meaning making .

The study was originally designed to investigate the nature and processes of digital and multimodal reading practices as experienced through touch pad technology. The specific research focus investigated how students’ reading practices varied across digital and print forms of text in terms of modal complexity and student perceptions of literacy practices within the classroom learning context. As Jewitt (2009) has shown, multimodal analysis is a new and somewhat contested area of research. Several researchers (e.g. Kress et al. 2001; Flewitt et al. 2009; Crescenzi et al. 2014) have used varied approaches to transcribing and analysing multimodal data in order to represent the relationship between different modes and meaning making. Methods of analysis use images, graphics, diagrams and tables to represent the impact of modal interactions on semiotic exchange. Influenced by these researchers we have previously used diagrams and tables to examine the relationship between modes and students’ learning responses and social interactions (Simpson and Walsh 2014). In this chapter, we are further testing the potential of such a framework focusing on the interaction of modal affordances in public and private learning spaces .

Participants in the ethics-approved study were twenty-eight Year 5 (aged 10–11 years) students in an urban independent school for boys in Sydney, NSW Australia. The study ran for three school terms, approximately 7 months. In the data collection phase, two researchers visited the participating class once a week during morning literacy sessions to observe the 28 students and their teacher. Our qualitative methodology prompted the collection and coding of different data sets in order to address our three focal points: 1. Literacy practices ; 2. The modal affordances of literacy practices using iPads; and 3. The classroom context in terms of zones of interaction. This paper provides exemplars representing the three foci as viewed through a methodological perspective attending to mode and learning purpose. Data were collected as video recordings of students reading print and touch tablet texts, artefacts in the form of work samples created during lesson time , fieldnotes of whole class lesson observations and pair or individual student think alouds. Think alouds are a method of data collection, which prompt students to reflect orally on their learning either during or shortly after a lesson takes place. The spoken thoughts are recorded by the researcher and transcribed for analysis. The researchers found think alouds to be a particularly robust source of information especially when students observed videos of their own actions and explained the impact of modes on their learning . The way these data were collected provided additional insight into the complexity of students’ learning helping the exploration of multimodal layering .

To illustrate student learning over time a sequence of learning activities, run over a number of days, has been selected for descriptive analysis. The end goal of the set of literacy tasks was for pairs of Grade 5 students to produce a persuasive text in the form of a radio advertisement about the dangers of junk food and to present this to the whole class. Throughout the sequence the teacher taught the students about the purpose, structure and language of persuasive texts by modelling successful texts and providing them with resources such as factual and persuasive texts about junk food in print and video forms. The students accessed these resources on their individual iPads as well as viewing them on the IWB during whole class discussion. Students were required to discriminate between opinion and fact through deconstructing texts and writing their own phrases and sentences. There was a constant shift between whole class work and individual/paired work with the texts on the students’ iPads. By the end of the lesson sequence the students had planned and written their persuasive texts and produced them as radio ads using multimedia for recording and editing with GarageBand on their iPads.

Framework for Analysis

As this chapter concentrates on exploring the concept of multimodal layering in terms of private and public learning spaces , our data analysis mainly addresses the dual focus of the individual student/learner and the classroom as a social context . The video recordings of students working in the classroom were analyzed by modal affordances and zone of social interaction . To make this kind of analysis possible we devised a methodological template, or framework of analysis, which attends to individual modes as well as the meaning making opportunities created from interrelated modes employed to achieve literacy tasks.

The three focal points were included in the framework for analysis of student learning events and represent the decisions we made as researchers to attend to:

  1. 1.

    Literacy practices = the explicit literacy focus the teacher introduced in a learning event

  2. 2.

    Modal affordances  = the interplay of semiotic systems including but not limited to digital platforms such as iPads

  3. 3.

    Zone of social interaction  = the interconnections of private and/or public spaces where student attention was directed by the learning context

The three headings relate to our interest in examining how the design of classroom contexts can construct opportunities for students to engage in multimodal digital practices that improve their literacy capabilities and contribute to critical reflection on learning .

Exemplars with Descriptive Analysis

The exemplars were selected from the data corpus as they track one pair of students who collaborated on the creation of their persuasive text from beginning to the end. These students were the only pair to take part in the reflective interview process watching a video of themselves working on their iPads. Each exemplar provides a progressive snapshot across the learning sequence that demonstrates how the multimodal layering builds up. The exemplars illustrate the way we have used our framework to analyse the interrelationships between literacy learning demands and modal interactions that were negotiated as students moved between private and public learning spaces in the classroom to achieve the literacy tasks. In each example, we introduce the classroom learning context and juxtapose it with a summary of what occurred captured in table format. A short transcript extract of student language is included in each exemplar where relevant to demonstrate students’ thought processes as they worked with the multimodal tasks. Included at the end of each exemplar is a comment about evidence of student learning. We demonstrate through this descriptive analysis how our concept of multimodal layering aptly captures the dynamic learning processes in which students were engaged. It should be noted that the distinction of ‘zone of interaction’ focus is viewed as more of a cline rather than either/or. After considering these snapshots of data capturing the social dimension within which the individuals act as learners, we will discuss the impact of multimodal layering on student learning.

Screenshot images for each Exemplar are included in Fig. 5.2 to show the shifting focus of attention from meaning making with ‘isolated’, private texts to intrapersonal meaning making with texts shared communally in public spaces. The images are positioned in Fig. 5.2, according to their semiotic reconfiguring in relation to Multimodal Layering .

Fig. 5.2
figure 2

Illustrated example of semiotic reconfiguring as multimodal layering

Full size image

Exemplar 1: Reading Phase

See Table 5.1.

Table 5.1 Multimodal layering at reading phase
Full size table

In the reading phase of the sequence the boys read information to themselves. There was no discussion and no writing. The researchers did not interrupt the students’ silent reading so we only have the images as verification of the individual, contemplative work they were doing. Our observations note students attended to and interpreted text elements, as they were required to discriminate between fact and opinion at an individual level of understanding. From the range of examples students read and viewed on paper and screen , they were given opportunities to identify how words, visual and also sound effects were used in persuasive texts compared with factual texts.

See image 1, Fig. 5.2 Screenshot of student reading fast food facts on iPads.

Table 5.2 Multimodal layering at writing phase
Full size table

Exemplar 2: Writing Phase

See Table 5.2.

During the writing phase of the sequence the boys reflected on their thinking as follows:

R:

Why did you start the sentence with I think?

S1:

Oh because it is an opinion and then followed by a fact

R:

exactly

S1:

We’re writing I think McDonald’s gives out too many toys a year

[S1 waves hand for emphasis]

See image 2, Fig. 5.2 Screenshot of hand written draft of radio ad on butchers’ paper

Note the students’ use of thinking verbs and nouns associated with the concept of persuasion as well as recognition of mode of communication. As indicated by the bold typeface, the students were using tone to emphasize their opinion. Students had attended to and interpreted the link between the generic models and applied them to their own practice. The language on individual post-it notes shared on the IWB showed the similarity of persuasive expressions students had chosen to record. The shared notes led to more complex examples of persuasive writing when students collaborated in pairs.

Exemplar 3: Talking and Listening Phase (Private)

See Table 5.3.

Table 5.3 Multimodal layering at talking and listening phase (private)
Full size table

During the creation of the radio text the boys simultaneously created two texts—spoken and music—the overlapping lines indicate their shift of attention between purposeful choice of modes in the following transcript.

S1:

[starts to read text from sheet on the floor] You think McDonald’s is good for you but it represents… McDonald’s gives out way too many toys a year, if you don’t

S2:

[navigates to music part of the app]

S1:

have to OK? restart that

S2:

I’ve discovered a little piano—[he works on creating a music theme and saves it]

S1:

[resets S2 iPad GB to start recording]

S1:

We need to say a line each, We have to say a line each

S2:

[plays music theme from iPad]

See image 3, Fig. 5.2 Screen shot of GarageBand audio production

Note the way the students used language (e.g. ‘I’, ‘We’,‘need to’,‘have to’) to self direct themselves about the use of the modes as they were composing their GarageBand text. The students reinterpreted and reframed concepts from their original monomodal print text repurposing them through the use of synchronous semiotic modes and transforming them into a new multimodal text demonstrating cohesive construction of meaning.

Exemplar 4: Talking and Listening Phase (Public)

See Table 5.4.

Table 5.4 Multimodal layering at talking and listening phase (public)
Full size table

At the end of the lesson when the teacher asked the students to listen to a radio ad one of their peers read aloud, the students were given a rubric to assess text features such as cohesive argument, facts with supporting evidence, persuasive language and emotive vocabulary. The short text extract from the transcript shows a student identifying the language he heard in the radio ad. Then it shows the teacher reminding the students about the kind of ‘tone’ they were aiming for in order to amplify their persuasive language through voice.

Teacher:

what was very impressive about that group’s radio advert?

S1:

instead of just saying McD is very bad for you he said this is why I agree McDonald’s is.

Teacher:

Yes he does use those connective persuasive openers and those connecting phrases. Microphone to S2 please. What style is yours? dramatic group—Switch on to find out why the world is so bad (spoken in a dramatic voice)…

Note the way the teacher modelled the voice tone that he intended students to associate with a mode of communication for intended purpose and student reflection on grammar and tone. Students attended to and interpreted the modal elements used in the peer text discriminating the use of grammatical features and appropriate voice tone at an individual level of understanding prompted by an assessment rubric. They demonstrate well-developed textual awareness. It should also be noted that this transcript is evidence of how the facts that were read originally in the McDonald’s prompt sheet in Exemplar 1 became layered into the new audio text created on the iPad.

Exemplar 5: Critical Reflection

See Table 5.5.

Table 5.5 Multimodal layering at critical reflection phase
Full size table

In the final example analyzed the video shows a pair of students watching a video of themselves on an iPad . Therefore, in image 4 the screen shows two layers of multimodal interaction from the researcher’s perspective looking over the shoulder of the boys as they look over their own shoulders on the archived recording.

S1:

So at the start L was lost and … [gestures with hand in front of iPad]

S2:

So basically he was helping me [S2 points to himself using the iPad] to find what to do here, he’s just helping me [S2 points at S1 in iPad video] and then we record using that [points at specific icon on the GB screen visible on the iPad] function, the microphone

S1:

the audio recorder

S2:

the audio and then we just read what we had on the script [S2 uses broad sweep of his hand across the image of the butchers’ paper on the iPad] and record it [S2 gestures to the iPad he was using that shows him recording] and then add some music in at the start and at the end and yeh. That is what we did

S1:

and to make it a bit more fun we added some pictures [S1 points at the little cartoon on the butchers’ paper seen on the iPad]

See image 4, Fig. 5.2 Screenshot of students watching a video of themselves as they create their radio ad.

Note the use of verbs and nouns associated with the affordances of the tablet to achieve the literacy task as well as recognition of modes of communication. The viewing of a video of themselves enabled the students to provide a spoken reflection on their use of multimodal affordances . They demonstrated critical awareness of how they had deliberately chosen synchronous semiotic modes transforming them into multimodal texts for cohesive construction of meaning.

Coda

We present one more example of layering during the interview where one of the students spontaneously commented on the affordances of the iPad . This point is important as, unprompted, the student notes the difference between the physical action of writing on paper compared with the multiple actions available with a keyboard and tablet. Image 5 is a still image that captures his action of physically mimicking the action of writing as he states:

See image 5, Fig. 5.2 Screen shot during reflective interview

S1:

Like I said it [the iPad] has more options so as he was saying it’s, in a book you go write, write write (student uses similar gestures mimicking his peer’s action and pretending to write on paper) where in an iPad you say will I add this in or maybe I’ll leave that out and then I’ll add that later (student imitates the gestures of working on a keyboard and tablet going from different movements, actions). It’s just got more on it than in a book.

This comment reveals the student’s understanding of how modes impacted on his learning. The physical actions accompanying his speech underline his realization of the relationship of embodied materiality on immaterial meaning making .

Summary

In our analysis of this sequence of private and public literacy events sketched with exemplars shifting through reading, writing, talking and listening and viewing and representing we found there was often a pattern of learning that was interwoven with activities in such a way that multimodal affordances became a support to students’ meaning making practices (Rowsell et al. 2013). As we documented the constant to-ing and fro-ing between texts, devices and learners we observed what appeared to be instances of student insight into how the modal affordances of the iPad supported their learning. However, this intellectual exercise depends on the students being able to make complex conceptual connections that could be described as cognitively demanding. For example, at the same time we observed that accompanying these intellectual insights were instances of engagement conveyed through hand gestures (indicated in Exemplar 5) and tone of voice. These were evidence of affective responses that demonstrated the intensities of the students’ involvement in their learning (Leander and Boldt 2013). We find that the term ‘multimodal layering ’ best describes the outcome of these combined processes and actions as the different modes of learning became sedimented one over the other. The images and the transcript examples above demonstrate multimodal layering in action during the lesson sequence about persuasive text writing. The exemplars showed students engaged with texts at different metacognitive levels to make meaning as their focus was drawn from private contemplation to public sharing of texts. It is important to note that the conceptual understanding of students within modal layering was made possible by the way the teacher planned the learning tasks to provide a range of affordances in a range of interactive contexts.

Discussion

Our methodological framework of analysis enabled us to document students’ learning through the successive stages of the classroom tasks. As shown in our data samples taken from observation, video clips and talk alouds, through our analytic framework we were able to examine the impact of multimodal layering on student learning. It appears to us that the methodology enabled us to identify how the use of the iPad in this learning context led to complex meaning making and affective engagement. Close examination of classroom interactions using a framework for analysis such as this demonstrates the dynamic interrelationships and interdependence of modes (e.g. touch, sound, image, spoken and written language) within pedagogical contexts (Leander and Boldt 2013). This approach allowed us to acknowledge the complexity of modal layering created in the collaborative learning activities made possible as student attention shifted from independent private to teacher directed public spaces (Simpson and Walsh 2014). Although it is not a delicate instrument with timing and actions monitored closely (Crescenzi et al. 2014), the contribution of the methodological ‘framework of analysis’ to multimodal studies is that it displays the interactions between meaning making events. We deliberately did not stage our data collection to monitor students moment-by-moment preferring rather to capture the learning schematic and view the conceptual traffic through a modal lens. What this approach revealed was the ways in which learners attend to and cope with modal complexity . Rather than presenting a finalized topography of the learning, the approach examines the individual layers of learning processes as they build one on another. Students were observed reading and creating texts using a variety of modes on screen and in print within different learning contexts in the classroom. The framework allowed us to attend to the interplay of these material and immaterial learning spaces. We propose that by slowing down the action we are able to identify phases of conceptual development for students working individually and with others. In our previous papers (Walsh and Simpson 2013, 2014; Simpson and Walsh 2014) we focused on touch. In this chapter we have taken a broader approach to see how the iPad supported multimodal literacy learning to occur across a range of modes.

Our findings in this study have addressed the question, what is the impact on complexity of conceptual understanding when different modes are layered into a learning context? We have shown that dynamic materiality influences the blurring of ‘public’ and ‘private’ learning spaces to create greater opportunities for individual as well as collaborative learning. The use of iPads in this classroom enabled the constant shift between modes through which students needed to navigate to build cohesive layers of meaning between reading and writing for literacy and learning tasks at school. This modal layering prompts students to reconfigure their existing mental schema as comprehension plays out as socially mediated cognition (Cain 2010). We claim that there is evidence of students developing meta-awareness through the processes they used when they were composing their radio ad text. Their talk is peppered with references to thought processes as well as the purposeful use of specific modes, as shown in a students’ comment to the teacher:

S5:

Well instead of just saying like McDonald’s is very bad for you. He said like, (uses presentation voice) ‘This is why I agree that McDonald’s is very bad for you’.

Student think alouds provided us with verification or new interpretations of their learning. This one shows, for example the student’s awareness of how persuasive rhetoric can be enhanced by changes in wording, emphasis and tone.

Further, if we go back to the quotation from one of the boys shown in the Coda we can see the physical, cognitive and affective engagement of this student as he evaluates their work and comments on the differences between writing on paper and creating on screen . He physically uses his hands to tap on the desk as he says ‘in a book you go write, write write’ emphasizing a dominant tone in his repetition of the word ‘write’ as he taps. In contrast when he describes the increased options of the iPad he uses a gentler persuasive vocal tone as he imitates the gestures of working on a keyboard: ‘… in an iPad you say will I add this in or maybe I’ll leave that out and then I’ll add that later’. His explanation and accompanying actions reveal the multimodal layers of learning he has experienced from the initial task of reading and creating a persuasive text to reflecting on the process and affordances of different modes.

For these students learning was an organic process, as they worked conceptually and emotively through progressive tasks located at the intersection of private and public spaces . The data exemplifies how the affordances of the iPad help to constitute dynamic learning/teaching events. Leander and Boldt (2013: 44) question whether a teacher can ‘recognize differences, surprise, and unfolding that follow along paths that are not linear’. Our study has uncovered the layered nature of learning opportunities. We hope that our research can provide insights into the potential of multimodal environments, which could inform pedagogy and support student learning.

Conclusion

The iPad platform provided students with ways to explore modes, design , enhance and communicate their textual creations. The analysis of student data revealed the development of conceptual complexity growing in a cumulative fashion as the students moved from planning and creating a persuasive text to reflecting on their learning at the end of the process. By tracking from reading, through writing, talking and listening and onto representing and critiquing we were able to show how the modal demands were layered as texts were repurposed from context to context . We were also able to show that due to the semiotic processing required to deal with each new configuration of text, students became more aware of their meaning making choices. We suggest there was a symbiotic relationship between the digital and non-digital literacy practices supported through the iPad platform.

Results show students shifting across modes and dimensions of social complexity through dynamic multimodal meaning making practices. Our conclusion is that as the students in this class were making meaning in public and private learning spaces using iPads/tablets they needed the flexible ability to attend to, interpret and repurpose synchronous semiotic modes in what Bezemer and Kress called a ‘chain of materialization processes’ (Bezemer and Kress 2008: 172). Through the use of qualitative analysis, as read through the frame of multimodal theory, we have demonstrated how our working definition of multimodal layering enabled us to attend to the multiple demands made on student attention in terms of semiosis and cognition.