Keywords

1 Introduction

In Fall 2015, I worked for a state-funded English as a Second Language (ESL) program designed for refugee-background adults from a variety of educational backgrounds and literacy levels. The program was housed in an adult basic education program and had a community education feel with open enrollment and no attendance policy. One week, four new students with no prior experience with formal education joined my class. That very same week, I was instructed to administer a programmatic assessment to all my students. I was told the data from the assessment would be used for program metrics and would be sent to the funding agency. Because I had substantial experience teaching refugee-background L2 adult emergent readers, I was immediately concerned about how the assessment itself might influence the experiences of students who had varying literacy and formal schooling backgrounds. I was also worried about the content of the test—and what it could usefully measure. There were complex multimodal ensembles, and questions with content potentially more suitable for students enrolled in classes higher than a literacy-level ESL class. However, because I was a new teacher in the program and because I understood the program’s funding was partially dependent on metrics generated by test scores, I felt obligated to administer the test. I reluctantly gave the assessment, imploring students to do their best, not be nervous, and simply try. But I saw their faces. I saw their reactions. And the next day, the four new students did not return, nor the next, or the next. This event has profoundly shaped the core of my research agenda which critically examines the design of language and literacy assessments, particularly in-house language and literacy assessments. I know now that even when assessments are meant to be low-stakes, tests of any sort can have an effect on students’ personal well-being, and academic self-confidence and performance (see, e.g., Cassady, 2004; Stiggins, 1999).

With concerns and questions about the validity and utility of such assessments in mind, my research has endeavored to uncover and understand some of the ideologies of language and literacy, and assumptions about learning, that drive and shape the multimodal composition and visual design of language and literacy assessments for refugee-background adult second language (L2) learners with emerging literacy. Here, I report on key findings with the following focal questions as a guide:

  1. 1.

    What test genre elements are used in the design of language and literacy assessments?

  2. 2.

    What semiotic resources (multimodal components, multimodal composition) are used in the design of the assessments, and how are tasks and/or messages portrayed to test-takers via these semiotic resources?

1.1 Literacy

Stemming from theories of scholars such as those of the New London Group (1996), the definition of literacy used in this chapter is that literacy is embedded in the social context, is more than a set of decontextualized skills, encompasses more than the mode of the written word, and is concerned with what people do with texts in real-world contexts (see also Barton & Hamilton, 1998; Street, 1984). This definition incorporates the social semiotic aspects of images and textual design, and is inclusive of visual and multimodal literacy, which are the abilities to understand and interact with visual images, multimodal genre elements and components, visual and multimodal design, and the relationships of these non-(print-)word-based modes of communication to both each other and to written words (see Serafini, 2014).

1.2 Adult Emergent Readers

Adult L2 learners with emerging literacy are adults who are becoming literate for the first time in their lives, and whose literacy is developing in a language they are also simultaneously learning (van de Craats et al., 2006). In the United States, for example, their literacy is developing in English. Most are from refugee- or immigrant-backgrounds.Footnote 1 As adult emergent readers (Bigelow & Vinogradov, 2011), many are learning about sound-symbol correspondences, punctuation marks, orthography, how to read and use a print and/or multimodal text, etc., as well as learning about the visual and multimodal aspects of literacy and of texts (Altherr Flores, 2017, 2019, 2020). Because their emergent (print) literacy is developing in a language other than their first language, they have unique programmatic and pedagogical needs. However, while most adult emergent readers are entering into formal, school-based learning for the first time in their lives, this is not to say they are beginning learners; as adults, they are beginning classroom learners, that is, they are learning how to “do school”, but they are not beginning thinkers. Following UNESCO’s (2018) definition for adult literacy, L2 adult emergent readers are aged 15 and older. Most of these learners are served by community or adult education programs, though some secondary school students also belong to this population (Bigelow & Vinogradov, 2011).

2 Literature Review

While there is a small, but growing body of research on adult L2 learners with emerging literacy or who (have) experienced interruptions in formal education, much of it has examined psycholinguistic processes, cognition, and metalinguistic awareness (e.g., Kurvers et al., 2006; Tammelin-Laine, 2015; Tarone et al., 2009; Vainikka et al., 2017). There is very little research—particularly recent research—about visual/multimodal literacy, and more specifically what I have called the social semiotics of literacy with respect to this population (e.g., Altherr Flores, 2019, 2020; Bruski, 2012; Whiteside, 2008).

Early work in visual literacy with adult L2 and literacy learners focused on cognition and processing, centering on literacy as a neutral, stand-alone skill. This scholarship included empirical studies, literature reviews, and recommendations for teaching visual literacy and for designing materials for international development work (Haverson & Haynes, 1982; Hvitfeldt, 1985; Linney, 1995). Although researchers in the medical community have researched visual literacy in medical materials (e.g., Dowse, 2004; Hill, 2008), these works are largely concerned with adults with “low” literacy in their first language, with insufficient attention to adult L2 and literacy learners.

More recent literature reviews and pedagogical recommendations for working with this population have mentioned visual and multimodal literacy (e.g., Arbuckle, 2004; Bigelow & Schwarz, 2010; Hardy, 2009), however, research on visual and multimodal literacy, and textual design for adult L2 and literacy learners is particularly understudied, especially in assessment texts (Allemano, 2013; Altherr Flores, 2017, 2020). Because print materials and assessments for this population often rely heavily on multimodal or visual elements, and because of how assessments are used (Shohamy, 2001), it is imperative to critically analyze the design of multimodal assessment texts and to investigate what assumptions may have been made in their design as these assumptions may impact not only test-takers’ meaning-making, but also how test administrators evaluate student responses. This review of the literature therefore informs a newer direction of what literacy—inclusive of visual and multimodal literacy—is for these learners as it incorporates the social, specifically the social semiotic, aspects of literacy.

2.1 Theoretical Framework

This study was guided by the theoretical framework of what I have called critical multimodal social semiotics (Kress, 2010; Kress & van Leeuwen, 2006; Pennycook, 2001). Social semiotics is the study of meaning and its social dimensions, but also of the power and processes of signification and interpretation, and how societies and individuals are shaped by these processes (Halliday, 1978; Hodge & Kress, 1988). A multimodal social semiotic perspective is founded on the notion that the meaning of a multimodal sign is shaped by the norms and social rules that were operating at the moment of its creation (Jewitt, 2009). A critical multimodal social semiotic theory considers how semiotic resources carry intentions, ideologies, and assumptions. Such an approach goes beyond description as it analyzes not only multimodal texts, but also their role in creating, reproducing, and transforming social practices, and considers ways to solve problems and issues of inequality (Caldas-Coulthard & van Leeuwen, 2003).

Modes are material-semiotic resources that are available for representation; they are culturally specific and socially created (Kress, 2010). Examples of modes include image, writing, speech, gesture, music, sound, layout, etc. Each mode can express the same intended meaning as another mode, however, the realization of this meaning differs from mode to mode (Kress, 2010). All texts are inherently multimodal as no written text can be disassociated from the material it was written on, and no visual image can be disassociated from the colors used to create it (Kress & van Leeuwen, 2006). The present study focuses on the modes of still image, writing, and layout.

This research relies on Kress and van Leeuwen’s (2006) uptake of systemic functional linguistics (Halliday, 1978), specifically the textual, ideational, and interpersonal metafunctions in their grammar of visual design. The textual metafunction allows different configurations of multimodal elements to represent different meanings. This metafunction is concerned with information value (marked by Given/New composition where the Given is located on the left, the New on the right; Ideal/Real composition where the Ideal [the generalized essence of information] is at the top, the Real [the practical or specific information] at the bottom; and center/margin composition where the information in the center is most important); salience (marked by visual weight, e.g., size, sharpness, contrast); and framing (the manner in which elements are connected or separated in a text). Through the ideational metafunction, visual structures are categorized as narrative (concerned with actions and events, and composed of Actors, Vectors, Goals, Reacters, and/or Phenomena) or conceptual (representations of the essence of a participant, shown through Possessive Attributes [parts and/or identifiers] and Carriers [whole structures]). Conceptual structures can be embedded in narrative structures. The interpersonal metafunction shows the relationship between the represented participants of an image and the viewer. Finally, this research also follows Kress and van Leeuwen’s (2006) use of the Peircean (1867) icon-index-symbol sign mode typology where icons are signs with a visual resemblance to what is being represented, indices are signs that resemble something that implies a concept or an object, and symbols are visual representations that do not resemble what they represent. Thus, critical multimodal social semiotics offers a lens through which to study literacy from a broader perspective that includes visual and multimodal design. It highlights assumptions that may have been made in the design of texts for refugee-background adult L2 learners with emerging literacy, and suggests possible solutions to address such expectations.

3 Methods

3.1 Site and Text

My critical analysis of multimodal texts investigates the test genre elements (e.g., multiple-choice questions, fill-in-the-blank questions, etc.) used in language and literacy assessments, as well as the semiotic resources used in the design of assessments, and how tasks are shown to test-takers through such resources. Through this research, I seek to unearth inherent, underlying assumptions in multimodal test design.

With these questions as a guide, and drawing on the theory of critical multimodal social semiotics, I analyzed a set of assessment texts used in a local ESL/literacy program for refugee-background adults in a large city in southern Arizona. The program, housed in an adult basic education program in a community college, is funded through the Arizona Refugee Resettlement Program. On average, the program serves about 600 students a year; it offers four levels of classes (literacy, low-beginning, high-beginning, intermediate). Students enrolled in the program are originally from a variety of countries that include Afghanistan, Bhutan, Burundi, Cuba, Democratic Republic of the Congo, Eritrea, Iraq, Somalia, Sudan, and Syria, with most students coming from Africa or the Near East.

The text under consideration is a set of in-house assessments that was developed “many years ago” (program director, personal communication, February 24, 2016) by a (now-former) instructor in the program and more recently edited and redesigned by an administrative support assistant. Neither the instructor nor the administrative support assistant have backgrounds in assessment design or in assessment design for ESL/literacy learners. Due to a variety of institutional constraints, these less-than-perfect in-house assessments have been used over the years to assess the language proficiency and literacy levels of all incoming students and place them in the program’s various levels, as well as to assess the progress of all current students. Students are placed into a level based on how many questions are answered correctly; deeper analysis and/or more nuanced evaluation of student responses, in general, does not occur.

There are two assessments: a reading and writing test, and a speaking and listening test. Here I share findings from my systematic analysis of the reading and writing test (known locally as “the literacy test”). It is six pages in length and contains 26 questions. There is one form, ten multiple-choice questions, three fill-in-the blank questions, two short free writes, and one extended free write. Each genre element uses different multimodal components such as clipart images, photographs, lines, boxes, spaces, typed words, numbers, and punctuation, and differing multimodal layouts. Most test questions are demarcated by, at minimum, a number, while the composition and format of the answers varies per the different layouts and multimodal components used.

3.2 Procedure

I collected data by conducting a critical multimodal social semiotic analysis of the literacy test, informed by systemic functional linguistics (Halliday, 1978; Kress & van Leeuwen, 2006). Test questions were coded and then categorized by genre element type: directions, form, multiple-choice, fill-in-the-blank, short free write, and extended free write. Within each of the test question types (i.e., genre elements), the textual, interpersonal, and ideational metafunctions were investigated, as well as the multimodal components (e.g., boxes, bolding, left/right layout, images, words, etc.). The metafunctions are not seen as separate parts of a text, rather I view them as intertwined and interconnected aspects of a text that collectively shape a text’s meaning—both in its design, and in how readers perceive it. While each metafunction can be analyzed separately, it is important to consider how the various multimodal aspects of textual design work together, and to analyze their impact simultaneously.

In the following section, I discuss the findings for the categories of form, multiple-choice, and fill-in-the-blank as those test genre elements use a large number of multimodal components; these findings can be extrapolated to the short and extended free writes. Of particular importance is to keep in mind how multimodal components are used in test questions and answers, and the design assumptions that may have been made by the test designer.

4 Findings

It cannot be overstated that adult L2 learners with emerging literacy are developing literacy for the first time in their lives, and that this literacy is developing in a language which they are also learning. Learners from this population come from a variety of ethnic, cultural, and linguistic backgrounds; language classes for refugee-background students in particular tend to have a diverse student population. The following analyses are rooted in a recognition of this combination of adult emergent readers and a diverse population.

4.1 Form

The test’s first page is a form (Fig. 1). Forms, in general, are characterized by provided words and blank graphic devices (boxes, lines, etc.). This form consists of a set of directions, with 10 numbered questions aligned vertically under the directions. Each question has either one or more blank lines or a blank box, or a combination of these two graphic devices. The form also has a set of parentheses on the right side of all but one question.

Fig. 1
A form asks to fill out the details of your last name, first name, male or female, address, and more.

Assessment page 1

Full size image

As a text, the elements of a page cannot be decontextualized; all the elements need to be read together. To analyze them, however, it is necessary to discuss them separately. The directions for this form have two distinct parts that are meant to be read in concert. The first part is a blurry clipart image of a pen. The second is a set of words that say, “Fill out the form”. This multimodal ensemble is placed at the top and left of the page, making both the image and words salient per the WesternFootnote 2 top/down (Ideal/Real) and left/right (Given/New) visual design conventions (Kress & van Leeuwen, 2006). These conventions mirror Western reading norms of reading from left to right, and from top to bottom.

The clipart image is to the left of the words. The image shows a ballpoint pen with a pen clip, and three lines on a piece of paper. The lines are symbolic, representing either writing or movement of the pen, or both. The lines and pen are also indexical, pointing to an action the viewer should recognize, namely that some sort of writing should be performed, and it should be performed on some sort of paper.

The words “Fill out the form” are bolded. Bolding is a graphic means of showing the importance of the words, similar to how in the mode of speech importance is designated by speaking more loudly or changing intonation.

This analysis reveals an apparent expectation that test-takers have taken tests before and are familiar with the format of a test. This is shown in the design of the directions where it is assumed test-takers will understand them and subsequently know how to respond to the question prompts. Test-takers are assumed to be able to: recognize and understand the blurry clipart image of the pen and that it signifies the act of writing; read and understand the words “Fill out the form”; and recognize and understand both that there is a relationship between the clipart and the words of the directions, and also understand what that relationship is between the image and the words—namely that they are both signifying writing in different ways. Another apparent expectation is that test-takers will understand the words are bolded as a means of showing their importance, and also know that the image and words are placed at the top left of this page as a result of top/down and left/right reading conventions, which are Western visual design conventions. Finally, assumptions have been made that test-takers will: comprehend that this multimodal ensemble of image, words, and layout tells them directly to do something (and realize these instructions refer only to this particular page); recognize and understand what this action is (to fill out this form); and know intuitively they should fill out the form with their own information. Because the exact content of what should be filled out is not provided here, the final assumption is that test-takers understand what kinds of content will count as appropriate or legitimate answers. Thus, while the idea of directions seems relatively simple, through this critical analysis it has emerged that directions are a quite complex multimodal test genre element.

Regarding the actual questions of the form, each question can be recognized by the bolded number on the left (Fig. 1). Like the directions, the bolding of the number seems intended to highlight something important about the number itself. In this case it highlights that the number is different from the question following it, namely the meaning of the number has no effect on the meaning of the question, but serves to visually differentiate the various questions. The form also utilizes the left/right reading conventions and the Given/New concept to delineate the number from the question. Further employing these conventions, each question is composed of a word or short phrase followed by a blank line on the right. The blank line is a graphic device, which, through indexicality prompts test-takers to write a response in this one specific space. This indexicality, however, relies on knowledge of Given and New, particularly knowledge that the “question” is the Given, the “answer” the New, and test-takers need to produce this New on the blank line. Conversely, the empty space surrounding the blank line, and above and below the words, is space where no writing is expected.

The specific content of the New, however, requires test-takers to know and understand it is their own personal information being requested on this form. As described previously, the format of question-posing reflects assumptions about what test-takers bring to the activity of decoding the form and inserting new, but relevant, information into the form in the expected places. Yet there is no explicit direction to test-takers anywhere on the form that they are expected to provide biographical details and information. Based on my observations over time and across settings, adult L2 literacy students may be tempted to “fill out the form” by performing what they know about school-based literacy and copy the words on the left onto the blank line at the right (see Altherr Flores, 2020). While performing the task in this way would provide new information, and would uphold the norms of visual design, the content would be incorrect per the expectations of this literacy test. This could influence students’ English-and literacy-learning trajectories in negative ways if they are not explicitly taught how to determine if a task expects copying—an often-used activity in literacy-level classes, or if a task expects written answers pertaining to the specific individual. Lack of such explicit instruction and explanation could lead to frustration for students.

Finally, there are punctuation marks (specifically, parentheses) on the right side of the page at the end of every question. When juxtaposed to the question and answer as one multimodal ensemble, these empty parentheses represent the New. Here, the new information is the score a test-taker earned for every question. There is no indication, however, from the graphic resources, the writing, or the layout that test-takers should not make marks in this section, and that it is instead intended for completion by another person (the test administrator or absent test grader) in a different temporal sequence.

The format and content of the form’s questions reflect assumptions that the purpose of each question is self-evident, and the expectation that the act of providing answers is straightforward. Writing the expected answer on the blank line of this form, however, requires: recognizing that a relationship exists between the number on the left and the words, graphic devices, and punctuation marks that follow it on the right (because 1 is different from 2, it signifies that each question, and therefore each answer, is different); understanding the relationship between the words on the left and the blank line on the right; understanding the relationship between the blank line on the left and the punctuation marks further to the right, and the relationship of these marks to the rest of the page; understanding that something must be performed on the blank line and that a key to how to perform it can be found in the directions above; and understanding the implicit directive to provide specific types of personal information in response to various randomly-ordered abbreviated question prompts.

Test designers may use forms as a means of giving students the opportunity to write something they perceive to be “more authentic” than other test question types/content.Footnote 3 And yet authenticity is not the issue with a form. The issue is the kinds of complexity in a form—complexities which test designers have not considered and have overlooked in their eagerness to provide students an authentic task. Thus, the apparent belief is that because the task is authentic, it therefore is a good test format, and because the purpose of the questions is obvious, the tasks are valid and fair.

From this analysis of the test genre element of a form, numerous assumptions made by the text’s designer have emerged. The first is that the provision of an image of a pen writing will be beneficial to test-takers who cannot read the words “fill out the form”. This perceived helpful inclusion, however, only further serves to demonstrate ideologies of assumed background knowledge (based on presumed “universal” experience inclusive of visual, multimodal, and print literacy). Most importantly though, the assumption that undergirds the entire form is that test-takers recognize and understand the overall purpose of the form as well as the specific goal of each individual numbered item/question on the form—namely that they should be providing certain types of biographical information in the appropriate spaces throughout the form. This is not a task of handwriting or attention to detail and difference (i.e., correctly copying letters and words), though implicitly those are addressed in this assessment. Rather it is an assessment of one’s ability to understand the relationship of the directions to an entire page of text and blank lines, to decode and comprehend the words of each question, to produce one’s own information to answer each question, whether by memory or by identifying, transferring, and copying the information from an identification card, and to appropriately interact with all graphic devices and other written modes of communication. What appears to the text designer as a simple task, made “obvious” through a variety of multimodal components, is a quite complex task for adult L2 learners with emerging literacy, one which requires knowledge and understandings of specific literacy and educational conventions and practices.

4.2 Multiple-Choice

In general, multiple-choice questions are characterized by a question and a set of provided possible answer choices. Of the 10 multiple-choice questions on this test, a variety of multimodal components and forms of multimodal composition are used. While analysis of all the questions revealed rich findings, here I address just Item #11.

Item #11 is at the top of the second page (Fig. 2). Above this prompt is a set of directions with a stylized, clipart checkmark and the word “Answer”. These directions show similar assumptions in design as were discussed for the first page; one difference is the use of a checkmark, which indicates the manner in which the answer should be written.

Fig. 2
A page asks you to choose the correct answer for the questions. The questions are given with three options. What time is it?, What is the appointment?, Where is the letter going?, and what material is the shirt made of?

Assessment page 2

Full size image

Of the questions on the page, only Item #11 contains the graphic device of the box in which a checkmark can be written. This graphic device is indexical. It points to the specific location where such writing is expected to be done. This design is complicated, however, by the sets of double dashed lines on the page, which are used as dividing lines between Items #11–#12, #12–#13, and #13–#14. Because the answer choices to Item #11 contain boxes, and because of the checkmark in the directions, it can be assumed that test-takers are expected to write a checkmark in a box, and not write on the blank dividing line. The relationships, however, between the checkmark and the graphic devices are socially coded and constructed. What is to stop test-takers from writing a check to the right of the word “answer”, or near the displayed clock? This brings the analysis to the next point of determining how genre components of “questions” and “answers” are constructed through multimodal design.

In Item #11, the entire question and answer section is denoted both by the graphic device of the (dividing) lines, and the use of a bolded number. The words of the question, “What time is it?” are also bolded. The bolding signifies the importance of the question, and sets it apart from both the answer choices and also the additional components of the question, e.g., the image accompanying the words “What time is it?”. Additionally, the words of the answer choices, while bolded, are smaller than the words of the question. The bolding makes these words salient in comparison to the surrounding white space, however the smaller font size also makes the words less salient in comparison to the question.

The question-and-answer ensemble of Item #11 follows the Given/New layout where the question (inclusive of words and image) is the Given and the answer the New. The Ideal/Real layout is also utilized, where the question (words) are placed on a higher vertical plane than the answer choices.

Aside from assuming test-takers know the Given/New and Ideal/Real conventions, and how they could be applied to a multiple-choice question ensemble, there is also the fact that a multiple-choice question relies on knowledge of question-and-answer adjacency pairs, and the idea that only one of the provided choices is correct (in this example, “It’s seven thirty”). While adult L2 literacy learners are not entirely new to the idea of adjacency pairs since they are a commonly-used function of speech (e.g., Speaker 1: “What is your name?”, Speaker 2: “Zeyneb”), the notion of a question and an answer in written communication, particularly where the speaker, or asker of the question (in this case, the asker of “What time is it?”) is not present, may be a new genre element. This relationship of first a question—because from a structuralist perspective an answer means nothing without a question, and then an answer is thus represented through the layout of the question and answer and the multimodal components used to compose it.

Delving further into the multimodal composition, the question of Item #11 is composed of two components (words, image) and three modes (writing, still image, layout), which together form a separate multimodal ensemble, the meaning of which exists only in the specific configuration of these components and modes. The words here are the Ideal, with the image as the Real. Thus, the words are the generalized essence, and the image is the details. Without the image, this question could not be answered in a specific manner; that is, a person could provide an answer, e.g., 12:41, but the image provides the details necessary for an exact time, 7:30.

The ideational content and type of image in this prompt must also be considered. The clipart is iconic, resembling an analog clock, and, in this instance, indexical, pointing to the specific time that must be acknowledged to answer the question. Additionally, this image is symbolic as clocks are a modern representation of sundials, which are used to indicate the idea of time in general. To answer this question, test-takers must be able to understand that: this iconic image is a clock; through indexicality the image references both a precise time and a specific tangible item that exists off this piece of paper; and the symbolic nature of a clock is used to represent time, the underlying content of the question. Furthermore, this clipart is not an authentic image that students would see in their everyday lives, though the argument could be made that images of this sort tend to litter educational texts and thus students may have had some exposure to them in their ESL/literacy classes, depending on the texts used by their programs. Such exposure, however, is not guaranteed. The use of such imagery is more indicative of a larger, more pervasive, assumption that clipart images are universally known and understood, and are therefore beneficial and/or appropriate for use in classroom materials and in assessment texts.

Relatedly, the composition of the answer of Item #11 relies on two components (words, graphic device) and three modes (writing, still image, layout). The answer choices (“It’s six o’clock”, “It’s six thirty”, “It’s seven thirty”) are provided as the Given. The New is expected to be produced by test-takers through interaction (in this case, writing) with the indexical box. Regarding the answer choice content, there is an incongruity between the clock’s numerical numbers and the word forms of the numbers in the answer choices. This could have been done intentionally to assess if test-takers can make the connection between the number symbol and the written number, however such an implicit goal makes this question ever more complex. This incongruity and complexity brings into consideration the intent of the prompt. Is the prompt meant to assess the test-taker’s ability to read a clock? To match number symbols and written numbers? Both? These considerations of goal and task must be taken into account when designing assessments for the focal population because all higher-level uses of literacy require knowledge of these foundational skills.

This analysis highlights a number of inherent beliefs held by the designer about test-takers’ experience completing this sort of assessment with this genre element type. The fundamental assumption is that test-takers will recognize not only that there are different elements and components in this section of the text, and they are different from the form section (and the following sections), but also that the combination of these particular test genre elements and multimodal components entails a specific purpose and response. Specifically, it is anticipated test-takers know their answer response should be in reference to the information provided in the question, and should not be creative or individualistic (such as information they produce on a form), and they should instead select one answer from the offered choices. When testing experienced test-takers, it makes sense to ask and expect them to select just one out of multiple answers provided, but this task becomes much more complicated when the test-taker has emerging literacy or has not engaged in this sort of written assessment previously. As my critical analysis of social semiotics here demonstrates, test designers must attend to much more than the written word; they must also take into consideration issues of layout, form, and composition (e.g., the use of bold font, the use of numbering systems, images, spacings, and other visual indicators of meaning).

4.3 Fill-in-the-Blank

The defining characteristic of a fill-in-the-blank test question is the use of a graphic device, specifically a blank line, at the beginning, in the middle, or at the end of an already-provided sentence. The blank line is indexical, referencing the location where test-takers should provide an answer. The line’s length can also be considered a hint for test-takers regarding the length of the expected written word or phrase.

Item #23 is a fill-in-the-blank question in the middle of page 4 (Fig. 3).Footnote 4 It is below a set of directions that are set apart from the page’s other multimodal genre elements by the use of two thin double lines serving as dividing lines. Within these lines is a clipart image of a pencil tip, and the word “Write”. Because the two lines enclose a space comparable in height to a blank line on lined notebook paper, and because they encompass the pencil tip and “Write”, they could appear as if they are meant to index writing, not page division. Furthermore, it is not explicitly stated what should be written, or where the writing should occur. Thus, test-takers could copy the word “write” or other aspects of the text, write answers from other portions of the test, or write something else entirely; this writing could be done on these dividing lines, or elsewhere on this page.

Fig. 3
A page asks you to choose the correct answer for the questions. The questions are given with three options. What is this used for? this sign means?, How much money is this? complete the sentence, and what is the best price per pound? are the questions posed.

Assessment page 4

Full size image

While Item #23 is located directly below these multimodal directions, it also has an additional subset of directions, “Look at the map.” which complicate the design and layout of this page and the items within it. “Look at the map.” functions as both directions and as part of the question of this question-and-answer ensemble. More importantly, these directions contain a stylized word, “look,” where clipart eyes replace the two letter “o”s in the word. These images are iconic: they look like eyes; indexical: they refer to an action test-takers should do, namely to look at something—this something is referenced in the second part of the sentence (“at the map”), but also assumed to be understood by viewers per the Ideal/Real layout with the something located in the Real position (below the eyes); and symbolic: the eyes replace the letter “o.” This symbolism is also simultaneously iconic because the eyes are in the shape of the letter “o.” This stylization, while it may be intended as support for beginning L2 learners, may negatively affect the meaning-making processes of students with emerging alphabetic and visual literacy, particularly students who may not realize that round objects such as eyes can stand in the place of letters in words. This iconic-indexical-symbolic stylization adds another layer of signification to the acts of decoding and comprehending a text. These layers could be envisioned as perceived helpful hints for test-takers, however their inherent complexity could be disadvantageous for test-takers to understand and accurately complete the task at hand.

Regarding the image of this fill-in-the-blank question, the focal assumption under examination is the belief that numerous modes and multimodal components can be combined to create an easily recognizable visual representation of a place from an imagined bird’s eye view. This assumption must be denaturalized as such combinations, particularly ones which utilize clipart, can be understood in multiple ways by students with varying experiences with literacy and education.

The image in Item #23 is a map with a store, a park, and (named) streets. The image is a conceptual image. It is an analytical process where the parts fit together to make up a larger whole, in this case, the larger whole is a map (Kress & van Leeuwen, 2006). This map is formed of three smaller analytical processes: the processes for the store, park, and streets. Via Possessive Attributes (Kress & van Leeuwen, 2006), the store is identified by the store signs, store shelves, light poles, and trash can; the park is identified by the bench and tree; and the streets are identified by the lines and associated street names. The neighborhood map is subsequently identified by these three items. Reading all these (smaller) identifiers in this image, however, requires culturally-specific content schemata, and this is in addition to the assumed visual literacy of knowing how to read a map, which is a specific type of conceptual image. Finally, there is an expectation that test-takers know and understand the abbreviations “Ave.” and “St.” Because the park is on 35th St., and the answer has “street” provided in the end of the sentence, knowledge of what “St.” means is important to prevent test-takers from writing “35th St.” in the blank, thereby making their answers read “The park is on 35th St. street.”

Like the multiple-choice question discussed above, this type of question also relies on an assumption that test-takers recognize their answers should be in relation to the additional information provided in the question section of the question-and-answer ensemble. Here, this additional information is the image, which is anticipated to be able to be understood by test-takers. This conceptual image, however, is laden with details and conventions specific to particular literacy and lived experiences, and for test-takers, reading such content as presented in such a complex semiotic manner may be as new for them as is the act of taking a test. Furthermore, it is potentially presumed test-takers understand the word they produce must fit syntactically into the sentence provided. This restriction, at first glance, may seem beneficial for test-takers as just one word is expected, as seen in this example, but it complicates the response test-takers can provide as a response with extra words will negate the correctness of the answer. Thus, the expectation that test-takers can read the entire sentence, imagine what word belongs in the blank, and then produce that word with no or little scaffolding is revealed, through this analysis, to also be quite difficult.

5 Discussion

The study’s results illuminate assumptions of visual/multimodal literacy, multimodal design knowledge, test genre knowledge, and assumed content and referential background schemata in the design of this in-house language and literacy assessment. There is an apparent expectation that test-takers already know what to do with this test, how to read it, how to interpret it, and how to respond. In short, test designers expect and assume that test-takers will bring knowledge about how to take a test as well as a nuanced understanding of the kinds of content that are relevant when providing answers. Such expectations assume that test-takers will know how assessments of this sort are used and evaluated, and what is done with the data from these assessments. These assumptions are undergirded by a belief that images, visual cues, and visual and multimodal design are universal or innate, when in fact these are embedded in the social context, and learned through experience and interaction. As Schneck (2005) noted, people are born able to see, but the understanding of the visual images that surround them is a product of learning and habit.

This critical analysis demonstrates that texts involve reading images and layout every bit as much as reading words. It is therefore imperative for text designers to be critically aware of their design in regards to visual and multimodal design. This has implications for assessments of any sort, from in-house classroom assessments with low stakes, to tests with higher stakes such as the U.S. naturalization test, to study materials associated with assessments, and more broadly to the design of texts of all sorts (including signs and handouts used in institutional settings). Thus, there are implications for educational institutions (K–12, adult, community), and makers of standardized tests like CASAS, TABE, and CAL,Footnote 5 as well as for the wide variety of institutions that serve and work with this population in other capacities (e.g., health, community, and non-profit organizations, workplaces, refugee resettlement agencies, the International Organization for Migration, the United Nations High Commissioner for Refugees, and governmental agencies like the United States Citizenship and Immigration Service).

These results also yield implications for teaching, specifically for educators teaching adult L2 learners with emerging literacy, but also for teaching any population in any context. Educators need to critically reflect on their own teaching and materials and consider the design assumptions they may be making for in-class language and literacy activities that are of the formative, as opposed to the summative, type. Additionally, it would be of benefit to include and/or embed: (1) visual and multimodal literacy (e.g., what bolding, underlining, indexical boxes, indexical lines mean; how to read a visual image; how to read top/down and left/right layouts; how to understand a connection between multimodal ensembles; how to read clipart, line drawings, photographs) (see Britsch, 2009; Hecke, 2015; Royce, 2002 for discussions concerning general populations of L2 learners); (2) test genre knowledge (e.g., directions vs. multiple-choice vs. fill-in-the-blank; how to respond to a fill-in-the-blank); (3) text genre knowledge (e.g., posted sign vs. take-home letter); and (4) information about how to use a text as well as how texts are used by others, in classroom instruction for adult L2 learners with emerging literacy. Research suggests that approaches to language teaching that emphasize diversity—of languages, perspectives, and modalities—such as a multiliteracies approach (Cope & Kalantzis, 2000; New London Group, 1996) could support this population. This approach has been used in education and foreign language studies (see Paesani et al., 2015), and would be beneficial for this population of learners as well.

6 Conclusion

Returning to the vignette that began this chapter, we must consider how the four new refugee-background students may have reacted to the assessment had it not contained such complex semiotic resources. If the tasks had been more transparent through, perhaps, the use of photographs instead of clipart art, if abbreviations had not been used, if answers could be circled instead of checked, if the pages could have been less cluttered by the removal of dividing lines, unnecessary directions and/or examples, or question-and-answer ensembles presented with more space between them, would the students not have been so overwhelmed by the educational experience of taking a test—that is, reading and writing on pieces of paper with no assistance from their instructor or their peers—during their first-ever week of formal schooling (in a new language) that they would have returned the next day? How can tests be designed to be more accommodating for not only L2 adult emergent readers, but also for (recently) resettled refugees who are undergoing a host of new, potentially stressful, experiences? In light of the scarcity of research regarding both assessment design, and visual and multimodal literacy, but also the intersection of these two fields with respect to (refugee-background) adult L2 learners with emerging literacy, additional research is needed to answer these questions. Namely, how do students from this population of learners understand and engage with different configurations and compositions of various multimodal components in assessment texts (see Altherr Flores, forthcoming)?

The results of this study are useful for designing assessments and texts for adult L2 learners with emerging literacy, but they also raise questions for educators (broadly defined) and text designers creating any sort of materials in any context. What other types of assumptions may text designers be making, particularly in the design of materials for readers who may be entering into new literacy practices? The research calls for a deeper level of awareness of semiotic resources in designed texts for the specific population of refugee-background adult L2 learners with emerging literacy, and hints at implications for broader populations as well.