Keywords

1 Introduction

Although important strides have been made in understanding bilingual youth’s multimodal composing processes and products, study findings are often presented from a researcher’s or teacher’s gaze (see Smith 2018). A complementary view of students’ experiences with creating digital products in schools from their perspective is also needed for understanding the affordances and constraints of conveying meaning through multiple modes. These student insights can be beneficial for effectively integrating digital multimodal projects in the multilingual classroom to support academic learning. Student perspectives offer valuable implications for the teacher’s scaffolding process by contributing to an understanding of how they conceptualize content, communicate their ideas, and infuse their identities while multimodal composing. Additionally, teachers can understand and address stumbling blocks students might encounter.

The handful of studies that have specifically examined bilingual students’ perceptions of digital multimodal composing illuminate how they positively perceive the multiple points of entry offered by communicating through visuals, sound, text, and movement for communicating complex ideas and aspects of their identities (DeJaynes 2015; Jiang and Luk 2016; Smythe and Neufeld 2010). In particular, this research demonstrates how multimodal composition fosters positive identity expression for Latinx and newly arrived immigrant students (Streng et al. 2004; Vinogradova et al. 2011) as well as the ability to shape with agency how they are positioned to others (de los Ríos 2018; Ivković 2019). These insights into students’ perspectives are often captured in out-of-school contexts (e.g., Omerbašić 2015), and more research is needed to understand bilingual students’ viewpoints and experiences with creating a wide range of digital projects in schools for academic purposes.

To address this need, we examined the viewpoints of 98 10th-grade bilingual adolescents who participated in three multimodal instructional units in an English Language Arts (ELA) class. Based on design interviews, written reflections, and video observations, this study presents the main themes of students’ perspectives when considering the learning potential and challenges of multimodal composing in the multilingual classroom.

2 Theoretical Framework

This study was guided by a social semiotics (Kress 2003, 2010) view of multimodality that emphasizes how various modes—including, but not limited to, visuals, sounds, text, motions, and gestures—are integral in meaning-making. Meaning occurs through the complex interaction between different modes, and the unique interweaving of modes communicates generative messages that no single mode communicates on its own (Jewitt 2009). Orchestrating multiple modes can create distinct opportunities for multilingual students to leverage cultural and social capital (Ajayi 2015; Bailey 2009), to express identities in ways not typically afforded by written texts (Cimasko and Shin 2017; Cummins, et al. 2015; Hull et al. 2010), and to “braid” home literacy practices with school practices to craft and develop multilingual narratives (Noguerón-Liu and Hogan 2017; Zapata 2014).

Central to a social semiotics perspective is the understanding that each mode is comprised of its own semiotic resources for communication. These unique modal affordances—based on their social histories, cultural uses, and material features—offer potentials that render specific modes more suitable than others for certain communicative tasks (Kress 2010). As described by van Leeuwen (2004):

Semiotic resources are the actions, materials and artifacts we use for communicative purposes, whether produced physiologically—for example, with our vocal apparatus, the muscles we use to make facial expressions and gestures—or technologically—for example, with pen and ink, or computer hardware and software—together with the ways in which these resources can be organized. Semiotic resources have a meaning potential, based on their past uses, and a set of affordances based on their possible uses, and these will be actualized in concrete social contexts where their use is subject to some form of semiotic regime. (p. 285)

Research describes how bilingual students often demonstrate and express different modal preferences for how they choose to communicate (Smith 2017; Smith et al. 2017). For example, one student might be able to express personal emotions visually in a way that is not possible through writing, whereas another student might prefer to rely on the specificity of the linguistic mode to convey their message. Some researchers have highlighted how bilingual students perceive different modes having different communicative affordances, including sound, visuals, and movement (Ajayi 2015; Ho et al. 2011; Kim 2018; Skerrett 2019; Smith 2019).

In relation to the current study, we are interested in how students identify the affordances and constraints of different modes for communicating their intended message and supporting their learning in an ELA classroom.

3 Related Research

Previous research on multimodal composing emphasizes the benefits from researcher and teacher viewpoints but has less frequently explored students’ perspectives. In first language (L1) contexts of English Language Arts settings, some of the few studies examining viewpoints demonstrate how students explained their specific modal choices while composing (Dalton et al. 2015; Jocius 2013; Smith 2017). Students have described feeling affirmed in their identities as multimodal composers (Dallacqua 2018), even if they have previously been positioned as low achievers (Jocius 2017). By seeking student perspectives, some research have further indicated how the flexibility of multimodal composing acts as a bridge for students in L1 contexts to integrate their out-of-school literacies into their academic learning (Taylor 2018).

In bilingual settings, research emphasizes the multiple benefits students perceive for composing multimodally. Students have expressed their views of composing as a means to affirm their multilingual and multicultural identities (Cummins et al. 2015; DeJaynes 2015; de la Piedra 2010; Vasudevan et al. 2010), discuss critical community issues (Amgott 2018; Anderson and Macleroy 2017), share their work with a wide audience (de los Ríos 2018), maintain their home country language (Omerbašić 2015), and rehearse while language learning (Jiang and Luk 2016). The auto-ethnographic nature of many multimodal projects has facilitated bilingual student connections to a larger heritage community, while also leveraging their identity reflections to envision their future selves and identities (de los Ríos 2018; Kumagai et al. 2015). Further, student perspectives reveal feelings of empowerment by sharing their out-of-school identities and experiences with peers (DeJaynes 2015), which has often facilitated discussion and action on local issues (de los Ríos 2018; Goulah 2017; Honeyford 2014). Students additionally described being encouraged through composing that involves collaborating with their families, heritage languages, and cultures to decolonize and restore the agency of their peers’ perceptions of their countries and communities (Cummins et al. 2015; Pacheco and Smith 2015).

Linguistically, bilingual students’ insights emphasize the importance of multimodal online spaces for linguistic and cultural growth. Learners of English commented that multimodal projects support an iterative composing that involves reviewing and correcting their language mistakes, and that this challenge motivates them to spend more time practicing English (Jiang and Luk 2016). Multilingual students have further conveyed the affordances of multimodal composing for translanguaging—or making use of multiple languages, registers, and/or varieties—to communicate with different audiences concurrently or to express themselves as individuals with dynamic identities (Kim 2018; Pacheco and Smith 2015). Translanguaging while multimodal composing allows heritage bilinguals and language learners to leverage non-linguistic modes to promote cross-cultural and linguistic connections (Anderson and Macleroy 2017; Kumagai et al. 2015).

4 Methods

Building upon this research, we examined the following guiding research question and sub-questions: What are bilingual adolescents’ perspectives on multimodal composing in the ELA classroom?

  • What do students view as the affordances of communicating with multiple modes in the ELA classroom?

  • What do students view as the constraints of communicating with multiple modes in the ELA classroom?

4.1 The Setting and Participants

This study was conducted in four 10th-grade English Language Arts classes at an urban Title 1 charter high school in a major southeastern city in the USA. The school was situated in a community composed of Cuban exiles and families who immigrated from Central and South America. Out of the 98 participating students, 96% had a heritage language other than English (Table 7.1).

Table 7.1 Self-identified Demographics of Students (N = 98)
Full size table

4.2 Multimodal Composing Units

Students participated in three multimodal composing units throughout the school year. All students in the four classes were invited to participate in the study, and those who declined were still able to engage in the planned multimodal curriculum. The first unit was a four-and-a-half-week poetry unit at the beginning of the fall semester. The culminating project centered on designing a hyperlinked PowerPoint that analyzed the multiple layers of meaning in a poem. Students were provided a handout with 13 poems that represented a range of authors (e.g., Langston Hughes, Maya Angelou, Pat Mora, Walt Whitman, Emily Dickinson). Each poem uniquely connected to the unit theme of identity, including engaging with issues of biculturalism, racial identity, belonging, and/or immigration. The second unit was three weeks in length and occurred at the beginning of the spring semester; it involved students creating a persuasive podcast on a controversial topic (e.g., wall at the Mexico border, gun violence in schools, etc.). The third unit was at the end of the school year and four weeks in length. Students composed a video that explored a literary theme from Kurt Vonnegut’s short story Harrison Bergeron (1961). Selected by the teacher, this satirical dystopian story occurs in the year 2081 and describes a society where citizens wear handicaps to promote “equality” (e.g., the beautiful are forced to wear masks and the strong carry weights).

For each of the units, students participated in a multimodal composing workshop (Smith and Axelrod 2019) intended to cultivate intentional designing for targeted purposes and audiences. The units followed a similar scaffolded sequence that involved explicit instruction, combined with opportunities for students to analyze a variety of examples, receive peer feedback, reflect on their process, and follow their own unique modal preferences (Smith 2017; Smith et al. 2017). Students could choose their composing tools and with whom they collaborated for each project.

4.3 Data Collection

Multiple data sources were collected to gain bilingual students’ perspectives on their experiences with digital multimodal composing in the classroom. All 98 students completed a written reflection after each of the three digital projects (294 total reflections). In these, students answered a variety of open-ended and Likert-scale questions about the topics, including the affordances and constraints of communicating with multiple modes, how they analyzed literature through multimodal composing, and their collaborative composing processes.

In addition, 63 of the students participated in a 30-minute semi-structured design interview after at least one of the three multimodal projects (108 total interviews). Using a laptop that recorded the screen and audio, students individually pointed out elements of their work and explained the reasoning behind specific design decisions. Students also discussed their overall experience and views on digital multimodal composing in the classroom.

Finally, video observations were also collected for six small groups of students for each of the three multimodal projects (n = 27). Each small group also shared research laptops with screen capture software that recorded their composing activities during in-class workshops. This software tracked the movements of their mice, websites visited, and all media used and edited. The accompanying audio was also recorded during the composing process, which provided insights into verbal interactions. Time-stamped video logs were created for all screen capture files that recorded students’ compositional actions (e.g., image search, image design, audio search, audio remix, voice record) and collaborations.

4.4 Data Analysis

Qualitative data analysis (Glaser and Strauss 1967; Corbin and Strauss 2015) was an iterative process that involved three phases. First, each of the three student’s reflections and interviews was open coded to develop emergent categories. This phase involved identifying, naming, and categorizing the different viewpoints students shared on their experiences with multimodal composing in the classroom. During this phase, we regularly met to discuss and refine emerging categories on students’ perspectives. The second phase of analysis focused on refining the codes we initially developed and systematically developing relationships between them. This step involved organizing our open codes into categories and sub-categories. Again, we refined these categories across all of the data sources and discussed disconfirming evidence. For the final phase of analysis, we circled back across all of the data sources (e.g., interviews and reflections) to conduct selective coding (Corbin and Strauss 2015). We also examined the screen capture video logs to see if there were examples of each code from the process data. The goal for this final focused analysis was to validate whether our final overall categories were abstracted enough to encapsulate students’ perspectives for the variation in composers and digital projects. We worked to strengthen the trustworthiness of our findings by triangulating different sources and methods (Erlandson et al. 1993). We also strove to forefront students’ perspectives so that we could understand their experiences through their eyes.

5 Findings

In the following, main themes are presented focused on bilingual students’ perspectives on the affordances and constraints of multimodal composing in their ELA class.

5.1 Conceptualizing Through Multiple Modes

Students described how working with visuals, music, and videos helped them to conceptualize literary themes in the early stages of their composing processes. In many instances, meaning-making with non-linguistic modes often preceded students’ written notes or other textual aspects of their projects and provided a thematic foundation from which they constructed their analyses.

By conceptualizing through visuals, students collaboratively laid the analytical groundwork for their multimodal projects. This process involved conducting online image searches with abstract keywords (e.g., “culture” and “identity”). Next, students visually brainstormed by viewing and assessing multiple images produced from their searches. These viewings sparked generative conversations and connections to themes in the literature they analyzed. For example, Maddie and Isabella initiated their analysis of Langston Hughes’ “Harlem” (1958) by searching for keywords that “stood out,” including “dying dreams,” “forgotten dreams,” and “lost dreams.” With one side of their shared laptop screen displaying the poem and the other side designated for conducting searches, they engaged in productive discussions related to their search results and developing interpretations. Maddie described how working with images first aided her analytic process: “I thought putting images first would help me to understand the poem better with seeing it visually than just reading it. So, I thought we will put images first, so it would make it easier to find more literary devices.” Many students detailed following a similar strategy of using images as a springboard in their literary analyses. Some students also multimodally conceptualized themes by watching videos or listening to music at the beginning of their composing process. For example, Alvaro explained how music helped him “a lot to understand a theme” when interpreting Harrison Bergeron.

Conceptualizing through visual and aural modes also helped students to gain a sensory understanding of the literature, including being able to “see” the content. This pattern of being able to “visualize” literature through multimodal composing was echoed in several student responses. For example, Aurora explained, “I liked working on these projects because they made me ‘see’ literature and it makes me understand it better.” Examples of visualizing included students searching for and viewing images of difficult vocabulary (e.g., “imperialism”) as a means to better grasp the content to gain an effective understanding of different scenes originally depicted textually. Students revealed how experiencing the content through more than one mode challenged their initial readings of the text and to “think outside of the box and not stay so straight forward with [their] ideas” (Caleb).

5.2 Innovative Meaning-Making Through Multiple Modes

Students described how using multiple modes allowed them to “think in a different way,” which included expanding their options for communication. They detailed the ways in which they flexibly leveraged the unique affordances of specific modes to express meaning creatively, and many explained how they thought a specific image or song could better encapsulate their ideas than through writing alone.

With a broader communicative palette to work from, students often described specific modal preferences (Smith 2017; Smith et al. 2017) when creating their projects. For example, Claudia saw the benefits of using sound to persuade others for her soundscape on immigration: “I personally prefer sound because you’re more into it. It’s like a movie but in your head.” Javi expressed a visual preference for communicating his understanding of the main themes in the Langston Hughes poem “I Too”: “The pictures say so much, so when you take out the pictures, the text is kind of hollow.”

Students’ perspectives also demonstrated how they enjoyed being able to break free from the constraints of traditional writing assignments and to have more freedom in creatively selecting and combining different modes to convey their thinking. Across the final reflections, a majority of students explained how they favored the multimodal projects compared with traditional written essays:

Cuz with written assignment you have to have something specific—a body paragraph, your introduction, your ending. But in a video, it can be in a different order, and you can add different things.. .. It does not have to be one thing. (Li)

It helped because it allowed us to explore themes in a fun nontraditional way. We also got to do it the way we wanted, we got to explain things in our own ways. (Sabrina)

I prefer to analyze literature through multimodal projects… The traditional written assignments are more robotic and in multimodal projects I can be more creative and show more about what I think. (Liz)

In responses like these, students’ reasons for preferring multimodal composing ranged from having more flexibility, agency, and creativity in how they expressed ideas.

5.3 Identity Expression Through Multiple Modes

Communicating through multiple modes afforded opportunities for many students to express their identities in meaningful ways. Students explained how they were able to connect the ELA content to their bilingual and bicultural experiences, personal emotions, and out-of-school interests.

A theme throughout student perspectives was how they infused aspects of their own identities through their orchestrations of multiple modes. Mateo, for example, found ways to make linkages to the country he immigrated from when he was 13 years old when analyzing the poem “Legal Alien” by Pat Mora.

I was from Argentina and I moved here and some people thought, “Oh, he can’t hang out with us cuz he doesn’t speak English”… [referring to image] This is my Argentina’s flag. So, in this poem I felt related to the author cuz like I said, it [developing a bicultural identity] happened to me too. When I came here, first of all I considered myself Argentinian because everything was new, but now I consider myself American and Argentinian at the same time, so I feel related to the poem.

Other students echoed a similar design sentiment about being able to “show their individuality” and insert themselves “into the projects” by selecting personally meaningful visuals and sounds ranging from national flags, cultural songs, images of food or locations, and heritage language use. For example, Tara said, “Through visuals we can actually make ourselves a part of the story” when discussing her hypertext project on a poem by Maya Angelou. Sergio revealed how they selected a specific song for their video analysis project: “We picked that since it kind of relates to us—we are Cubans.”

Relatedly, students shared how using visuals and sounds allowed them to forge an effective connection to the ELA content. Brianna explained the emotional power of multimodal composing:

It [multimodal composing] gives more emotion compared to an essay. Although an essay is faster, this does give more emotion…I think people would be more affected through sound than an essay.

Angel also described his experience with creating the video theme analysis, “It’s different because you get to act it and feel the emotions rather than reading it…It gave us an idea of what it could be like to be in the story.”

Finally, students multimodally represented aspects of their out-of-school interests in numerous ways, including hobbies, skills, and relevant current issues. Students also described how they were able to make linkages between the class content to movies, television shows, music, and video games they enjoyed. For example, Celeste explained how her group was inspired by visuals and sounds in the television show, American Horror Story. She described in detail how they integrated similar “creepy” effects, including flashing lights, loud thunder, and masks on characters. Making these multilevel connections through interweaving media helped students “see how literature connects to real life” (Celeste).

5.4 Contextualizing Literature Through Multiple Modes

Students contextualized the literature by traversing networks of hyperlinked multimedia and informational websites to interpret the social and historical climate surrounding the work they analyzed. Many students explained how their process of contextualizing through multimedia aided them in “understanding” the literature, empathizing with the author, and “connecting the past to the present” (Amada).

To provide insights into the literature they analyzed, students referenced online multimedia resources when composing on their laptops. Instances of contextualization included reading informational web pages, watching related videos, viewing historical photographs, and listening to music from the time period. Contextualizing during their multimodal composing processes also offered students a window into authors’ lives and their possible exigence for writing. Many students also explored authors’ experiences and mental states—examples included Emily Dickinson’s isolation, Carl Sandberg’s experience in the military, and Maya Angelou’s sexual assault as a young girl—which offered an elucidating and empathetic lens for analyzing their poem.

Additionally, students described using their digital projects as a vehicle to share what they learned through their contextualizing processes. William, for example, considered his audience when incorporating historical photographs depicting culture in the USA during the 1920s, including jazz music from the Harlem Renaissance era in his analysis of Langston Hughes’ “I, Too” (1926) (Fig. 7.1):

Fig. 7.1
figure 1

William’s hypertext poetry analysis that Combined Photographs, Videos, and Music

Full size image

[W]e used different modes…Even though this poem did not literally give you the background of his time period, I think we kind of made it, so a reader who came and never heard this poem before would understand completely: “OK, this is who the author is, this is where he wrote it, this is when it was written.” We gave a whole basis; we made it into a movie, I think. We made the poem into a movie.

Figure 7.1 shows that William combined photographs, videos, and music to convey the historical context of the poem “I, Too” by Langston Hughes (1926).

Multiple modes and online resources mediated students’ processes by providing accessible ways to understand and share the context surrounding the literature they analyzed. As Maddie confessed to Isabella while composing their hypertext: “We are learning things I did not know!”

5.5 Constraints with Multimodal Composing

Along with sharing the affordances of multimodal composing, students also described constraints they encountered. The most common challenge revolved around various technical issues throughout their composing processes. Many students explained they had difficulty with some of the composing programs (e.g., iMovie, Audacity, and PowerPoint), including combining media, hyperlinking, and using different editing features: “Creating the video was harder than a written analysis. I don’t work well with all of those programs, so working with iMovie was difficult for me” (Elisa). Because of these technical stumbling blocks, students explained how they “had to ask for help a lot” from the teacher and their peers.

The second most common constraint with multimodal composing shared by students involved selecting “the perfect mode” to encapsulate their ideas. Steve explained, “I struggled with trying to find the right types of modes that sync really well with the theme of the poem.” Lena shared a similar constraint, “We struggled to find some certain images that represented what we meant and how we felt.”

Other constraints revolved around students not feeling they had enough time to complete their digital projects—especially if they did not have reliable access to a computer at home—along with confusion about exactly how the teacher wanted them to express ideas through multiple modes. A handful of students stated that they would have preferred to type their projects because they are more familiar with the academic writing medium and its expectations:

I personally prefer to write because I think it’s less work and it can be done in less time…Some of us are better writers while some of us are better are making multimodal videos. (Paula)

Finally, some students encountered difficulty when trying to make each group member’s designs cohere into one final project. “Putting it all together” was a challenge, as well as making sure their use of modes accurately represented their intended ideas.

6 Discussion and Conclusion

This chapter examined the perspectives of 98 bilingual 10th-grade students who created three digital multimodal projects in their English Language Arts class. Qualitative analysis revealed how students overwhelmingly viewed affordances of multimodal communication, along with some common constraints, when they were asked to communicate through multiple modes. These findings support and extend current digital literacies research on multilingual youth while also underscoring the need for eliciting student voices and supporting their multimodal composing processes.

The students’ view that multimodal composing afforded valuable opportunities to express their bilingual and bicultural identities and out-of-school interests echoes previous research (Smith et al. 2021; Yi et al. 2019). Multimodal composing offers multiple entry points and meaningful opportunities for multilingual students to connect to their lifeworlds and experiences while also working toward academic goals (Cummins et al. 2015; Honeyford 2014; Smith 2018). Furthermore, the students’ expressed a preference for digital multimodal composing over traditional academic writing aligns with previous research that describes how bilingual students are more engaged, motivated, and connected to their digital projects when they have more flexibility, agency, and creativity in how they express ideas (Jiang and Luk 2016; Goulah 2017).

This study also provides insights into how bilingual students were able to engage with academic content in new and innovative ways through their multimodal projects. Students described how they conceptualized literary themes through visuals and sounds as well as gained a unique sensory and affective understanding of the content. Through layering multiple modes, students made multilevel connections to “understand” the cultural and historical context of the literature they analyzed. These findings point to the potential for how multimodal composing can mediate learning in different content areas (e.g., de Oliveira and Smith 2019; Grapin 2019; Vandommele et al. 2017; Zheng et al. 2014).

Future research should continue exploring the possibilities of multimodal composing for mediating and transforming academic learning. As these findings are situated in a specific instructional context where students created three distinctive multimodal projects, much more needs to be understood about multimodal composing-to-learn (Smith 2019) across different composers, contexts, content areas, genres, and digital tools. It would also be beneficial if future research continued to examine how multimodal composing might support language learning and bilingual students in different stages of their academic, linguistic, and social development.

These findings also have implications for integrating multimodal composition into the multilingual classroom. As demonstrated, students voiced various constraints with multimodal composing ranging from technical difficulties, to making their collaborative projects cohere, and “finding the right mode” to accurately represent their thinking. A few students—particularly those who excelled at writing—were initially disoriented when asked to communicate their thinking through visuals, texts, sound, and movement. Although the classes participated in a scaffolded workshop model (Dalton 2013; Smith and Axelrod 2019), which involved explicit technical instruction, combined with opportunities for students to analyze a variety of examples, receive peer feedback, reflect on their process, collaborate, and follow their own modal preferences, some students still encountered various challenges. This finding underscores the importance of scaffolding students’ processes and the need for more work in this area. A related challenge is teacher preparation for these rich pedagogies. With growing number of teachers learning to work with bilingual students (Helman 2012; Lucas et al. 2008), research should prioritize investigating barriers for the integration of technology in the classroom and ways educators can effectively collaborate with students of varying linguistic proficiencies and schooling experiences (Ajayi 2010; Yi and Angay-Crowder 2016). Furthermore, it is important for educators to consider the specific affordances and constraints different modalities and digital tools offer bilingual students for meaning-making.

Finally, these findings emphasize the importance of valuing bilingual and immigrant students’ voices that are often marginalized. Students are frequently overlooked as stakeholders in education, although they are the principal recipients of curricular implementations like multimodal composing. It is thus crucial that educators and researchers center bilingual students’ experiences by listening to and learning from their perspectives.