Abstract
Given prior work identifying sourcing, corroboration, and contextualization as key strategies needed to learn from multiple texts, this study examines students’ multiple text processing and written response composition under three different heuristic conditions. In particular, students were asked to engage in sourcing, corroboration, or contextualization while completing a multiple text task. Then, log data of text access, students’ notes, and written responses were analyzed across task conditions. Moreover, each sentence included in students’ written responses was linked to information in notes and in the library texts provided. Results indicate that students directed to engage in sourcing both accessed more document information and included more of such information in their notes. More generally, students’ response composition was found to be reductive in nature, with students filtering information from library texts into their notes and into the written responses that they composed; at the same time, cross-textual integration was found to be limited.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
In a now seminal paper, Wineburg (1991) compared the reasoning of expert historians to that of Advanced Placement history students. He found experts’ historical reasoning to be characterized by their use of three heuristics, those of sourcing, corroboration, and contextualization. Sourcing refers to experts’ tendency to consider document information (i.e., text metadata like author and publisher) in interpreting texts’ content. Corroboration refers to experts’ comparison of texts to one another, while contextualization describes historians’ placement of texts in a temporal and spatial context. Since Wineburg’s (1991) initial work, examinations of students’ use of these three heuristics have been carried out primarily in the domain of history (Britt & Aglinskas, 2002; Nokes, Dole, & Hacker, 2007; Rouet, Favart, Britt, & Perfetti, 1997; Stahl, Hynd, Britton, McNish, & Bosquet, 1996), with some indication that these heuristics may be more broadly applicable as well (Bråten, Strømsø, & Salmerón, 2011; Goldman, Braasch, Wiley, Graesser, & Brodowinska, 2012; Shanahan, 2009; Stadtler & Bromme, 2013). Consistent with Wineburg’s (1991) initial work, students, even at the graduate level, have been found to rarely engage any of these heuristics, with corroboration and contextualization used least often of all (Nokes et al., 2007; Rouet et al., 1997). In this study, we build on this prior work in two primary ways. First, we examine students’ applications of the multiple text use heuristics that Wineburg (1991) identified beyond the domain of history, when students are asked to complete a multiple text task addressing a controversial topic in education (i.e., gifted education). Second, considering the infrequency with which students have been found to use the heuristics identified, we examine performance differences when students are explicitly instructed to engage in sourcing, corroboration, or contextualization during a multiple text task. More generally, we examine the associations among students’ text access, note taking, and written response composition during multiple text task completion.
Sourcing, corroboration, and contextualization
In the most comprehensive study following Wineburg’s (1991) initial work, Rouet et al. (1997) examined history (i.e., disciplinary specialists) and psychology (i.e., disciplinary novices) graduate students’ engagement in sourcing, corroboration, and contextualization when completing a multiple text task about the construction of the Panama Canal. Sourcing was assessed according to the number of references that students included in their essays composed based on multiple texts, with no differences found across graduate students in psychology vis-à-vis history. Corroboration was identified when students connected documents with similar points of view (i.e., positive connections) or identified documents as conflicting with one another (i.e., negative connections) in their essays, with some students linking groups of texts together more generally (i.e., general references). Nevertheless, Rouet et al. (1997) found graduate students, across domains, to include less than one instance of corroboration in the essays that they composed. Finally, contextualization was assessed by students’ provision of information based on their general world knowledge (i.e., general context statements), domain knowledge of history (i.e., historical context statements), or specific topic knowledge about the construction of the Panama Canal (i.e., problem context statements). It was here, in the contextual statements that individuals produced, that domain novices and domain specialists differed, with history graduate students including significantly more statements related to historical context knowledge in their essays than their psychology counterparts. As such, Rouet et al.’s (1997) investigation is important for at least two reasons. First, it provides a model for how sourcing, corroboration, and contextualization may be assessed in students’ writing, rather than during processing, as was done in Wineburg’s (1991) study. Second, it suggests that these three heuristics may be used not only by discipline experts to reason about history, but rather by discipline novices, as well.
Assessing sourcing, corroboration, and contextualization
While Wineburg (1991) used a think-aloud approach to identify experts’ and novices’ use of sourcing, corroboration, and contextualization, follow-up studies have favored trace-based and writing-based approaches to capturing students’ use of these strategies. Trace-based approaches focus on students’ processing during multiple text use. These approaches have examined various indicators, including computer log-files, data from eye-tracking, and students’ notes, for records of strategic processing when students learn from multiple texts (e.g., Gottschling, Kammerer, & Gerjets, 2019; Hagen, Braasch, & Bråten, 2014; Salmerón, Gil, & Bråten, 2018; Wiley et al., 2009). For instance, List and colleagues (List & Alexander, 2017, 2018; List, Alexander, & Stephens, 2017) used log files to capture the frequency with which students accessed document information (e.g., author background and credentials) when engaged in a multiple text task, while Britt and Aglinskas (2002) captured sourcing according to the volume of document and author information included in students’ notes.
Writing-based assessments of sourcing, corroboration, and contextualization have looked to students’ task products or written responses for evidence of strategy use. Sourcing has been examined most directly as reflected in students’ citation use during written response composition (Britt & Aglinskas, 2002; Du & List, 2020; List et al., 2017; List, Du, Wang, & Lee, 2019; Rouet et al., 1997; Wiley & Voss, 1999). Corroboration has been assessed both via students’ explicit juxtaposition of information from across texts (e.g., Rouet et al., 1997) and as reflected in students’ use of various connective terms (e.g., causal, temporal) to link information across texts (Britt & Aglinskas, 2002). Contextualization has been assessed by categorizing the statements included in students’ written responses as reflecting prior (i.e., contextual) knowledge (Rouet et al., 1997) or as transformed, corresponding to the combination of text-based information with information not from texts (Wiley & Voss, 1999). In this study, we use both trace-based and writing-based methods to determine the extent to which students engage in sourcing, corroboration, and contextualization, when explicitly instructed to do so during completion of a multiple text task.
Sourcing, corroboration, and contextualization beyond history
While the bulk of studies examining students’ use of these three heuristics have been in the domain of history, there is reason to believe that sourcing, corroboration, and contextualization may be more generally applicable strategies for learning from multiple texts. First, sourcing, or the evaluation of texts based on author and document information, has been found to be a key multiple text use strategy across domains (Bråten, Strømsø, & Britt, 2009; Bråten et al., 2011; Du & List, 2020; List et al., 2019; Thomm & Bromme, 2016). Stadtler & Bromme (2014) explicitly emphasized the importance of sourcing for learning in science. This is because students often lack the knowledge and resources necessary to independently verify scientific claims (i.e., make primary trustworthiness judgments); in such instances students are required to make secondary judgments about which author(s) to defer to or which sources to trust. These secondary judgments reflect sourcing or the evaluation of texts’ quality based on author characteristics. More generally, students’ engagement in sourcing through citation, or information attribution, is an academic practice widely used across domains (Burton & Chadwick, 2000; Hyland & Bondi, 2006).
Corroboration, as a multiple text use heuristic, has, likewise, been investigated across areas of study. Wiley et al. (2009) examined students’ multiple text strategy use during completion of a science inquiry task. In particular, students were provided with a set of seven texts, deliberately constructed to include some overlapping but otherwise unique information on the eruption of Mt. St. Helens. Although uncommon, when evaluating the reliability of the websites provided, students were found to engage in corroboration to at least some extent, describing websites that: “matched” and “kinda went along with” other texts (p. 1083). At the same time, echoing results from the domain of history, only 13% of students identified the corroboration of information across texts to be a method of determining the reliability of scientific information. In a follow up study, Goldman et al. (2012) found students performing comparatively better on a multiple text task to revisit more reliable websites, as compared to poorer performing students, a possible indicator of the association between corroboration and multiple text task performance in science.
In the area of literature, students’ intertextual reading, requiring corroboration, has likewise been examined as key to literary interpretation (Goldman, 2004; Hartman & Hartman, 1993). Many (1996), in a study of Scottish high school students, found them to engage in inter-textual strategy use across both literary and informational genre texts. For literary texts, in particular, students engaged in intertextuality to develop an enriched understanding of a given text (i.e., to understand an imaginary world more richly and deeply) and for personal acknowledgment (i.e., to form a personal connection to text content), with both of these processes considered to be associative, or corroborative, in nature. As such, across the areas of history, science, and literature, corroboration, intertextuality, and the juxta-positioning of texts in relation to one another have been demonstrated to be foundational skills for learning from multiple texts (Goldman, 2004; Goldman et al., 2016; McCarthy & Goldman, 2019).
Contextualization as a multiple text use heuristic has received perhaps the least attention of all, likely due to studies of multiple text use often involving low-knowledge samples, with students having limited contextual knowledge available to bring to bear on processing (Reisman & Wineburg, 2008; Van Boxtel & Van Drie, 2012). Indeed, students’ irrelevant or inaccurate prior knowledge engagement has sometimes even been found to hinder comprehension (Wolfe & Goldman, 2005). Nevertheless, given that prior knowledge has commonly been associated with students’ performance on multiple text tasks, across domains (Bråten, Anmarkrud, Brandmo, & Strømsø, 2014), we can assume that knowledge-based contextualization may help with inferencing and with the interpretation and evaluation of texts’ content across domains (Anmarkrud, Bråten, & Strømsø, 2014; Strømsø, Bråten, & Samuelstuen, 2003).
Throughout this earlier work, contextualization, or students’ prior knowledge use, was examined as a deliberate strategy, engaged by learners to support comprehension. Still, other work has suggested that the engagement of prior knowledge to validate claims in texts may also be an automatic process. In particular, the Two-Step Model of Validation (Richter, 2011; Richter & Maier, 2017) suggests that during comprehension students engage in two types of validation based on their prior knowledge (or beliefs). The first type, non-strategic validation, is automatic and tends to result in students’ construction of prior-knowledge-biased or prior-belief-biased mental models of the information presented across texts. This reflects students insufficiently considering conflicting or knowledge- or belief-inconsistent information during reading. The second type of validation, strategic elaboration, involves students’ deliberate reasoning about knowledge- or belief-inconsistent information and results in students constructing a more balanced representation of information introduced via texts, including belief consistent and belief inconsistent information.
In this study, we deliberately cued students to contextualize or evaluate information using prior knowledge, in the hope of engaging this latter, deliberate form of validation. Additionally, we chose a topic for our multiple text task that we expected students to be broadly familiar with. That is, we expected students to have both personal and academic knowledge of or beliefs about gifted education and we expected these to support students’ strategic elaboration. Indeed, Barzilai, Thomm, and Shlomi-Elooz (2020) recently demonstrated that students rely more-so on prior knowledge when evaluating information about a familiar vis-à-vis an unfamiliar topic.
Still, based on Richter and Maier’s (2017) work, we expected students, across conditions, to engaged in non-strategic validation, as a routine part of their reading process. We did not expect such non-strategic validation to necessarily be adaptive as it may have led students to ignore or neglect prior-knowledge- or belief-inconsistent information during processing, in favor of information that comported with their existing knowledge or beliefs (Britt, Rouet, Blaum, & Millis, 2019).
Topic-specific instantiations of sourcing, corroboration, and contextualization
In this study, we examine students’ sourcing, corroboration, and contextualization when learning about a controversial topic in education (i.e., gifted education). This topic was selected for three primary reasons. First, nested in education, this topic, allowed us to examine students’ multiple text reasoning beyond the realm of history. Second, this topic was complex, with a variety of stake-holders (i.e., parents, teachers, researchers) involved. This allowed us to attribute texts to a variety of differentially qualified and motivated authors and therefore to examine students’ sourcing when reading about a controversial topic across multiple texts. Relatedly, this topic was one which was multifaceted and controversial in nature, allowing both complementary and conflicting texts to be presented to students, facilitating cross-textual corroboration. Finally, this topic was selected because of the contextualization that it would allow. Specifically, this topic was expected to tap both students’ general, world knowledge about schooling in the United States and more specific, academic knowledge about gifted education. In particular, while we expected all students to have some general, contextual knowledge about schooling from their own experiences as students, we also knew that all of the students in our sample would have domain knowledge about education, and that a number of students would have taken specific courses in gifted and talented education, resulting in topic knowledge relevant to the task. As such, we expected this topic to allow students to engage in contextualization at all three levels of knowledge identified by Rouet et al. (1997).
In sum, the topic of gifted and talented education afforded us the opportunity to examine sourcing, corroboration, and contextualization as reasoning strategies within a novel domain context (i.e., that of education). Sourcing was expected to reflect students’ consideration of author and document information in evaluating texts. In particular, texts in this study were variably attributed to expert authors or to individuals holding applied expertise (i.e., a parent, and a teacher). Sourcing in this context was expected to result in students’ recognition of these author features and consideration of these features when evaluating and writing about texts. We expected the use of the sourcing strategy, in this case, to generally reflect what students are often asked to do when reading various perspectives in the popular press (e.g., students may elect to defer to research-based vis-à-vis applied points of view, or vice versa).
Corroboration, in this case, involved students comparing and contrasting information across texts. While Wineburg (1991) specifically examined corroboration as the comparing of facts and details in historical documents as a means of ascertaining their accuracy and veracity, we view corroboration as having both this local level purpose (i.e., verifying specific information), alongside a more global goal, wherein diverse perspectives are able to be juxtaposed across texts, with this more global connection formation better reflected in Rouet et al.’s (1997) work. To capture both of these aspects of corroboration, the texts used in this study reflected both overarching perspectives supporting and opposing gifted and talented education and specific information, able to be compared across texts (e.g., one text reporting that students learned better within homogenous ability groups, while another text presented evidence for heterogenous grouping).
Within the context of this study, we also viewed corroboration as a component of (although not synonymous with) integration. Corroboration was thought to reflect the explicit comparison of information across texts, whereas integration was considered to encompass both students’ connection formation across texts during processing (i.e., integrative reasoning) and the ultimate representation of texts that students formed after reading (i.e., integration performance as assessed through writing). While corroboration is a form of integrative reasoning, other types of integrative reasoning have also been documented (e.g., elaboration, or the use of one text to expand on information in another, comprehension support, or the use of information in one text to better understand another, Anmarkrud et al., 2014). In this study, we expected corroboration, alongside other types of integrative reasoning, to contribute to students’ integration performance or to their formation of a singular, coherent mental model of information presented across multiple texts.
Finally, contextualization in this study was viewed as students’ marshalling of prior knowledge or experience to understand and evaluate information in texts. We considered contextualization to be a highly domain general strategy, applicable beyond the domain of history. At the same time, we also expected prior knowledge to manifest in a somewhat topic-specific way in this study. That is, while the facilitative role of prior knowledge for reading may be considered to be a domain-general characteristic of contextualization, the type of knowledge engaged across domains may be somewhat different. Because students lack direct personal experience with history, Wineburg (1991) and others (Reisman, 2012) view contextualization as the placement of texts within a historical and temporal context, whereas, in this study students could bring to bear not only academic knowledge (e.g., about theories of learning) but also world knowledge and knowledge based on personal experience in reasoning about the texts provided. Indeed, work examining students’ reasoning about topics in education has often found students to rely on lay theories, rather than on academic knowledge in doing so (Fives & Buehl, 2012; Kendeou, Robinson, & McCrudden, 2019; List & Rubenstein, 2019). At the same time, Wolfe and Goldman (2005) in a think-aloud study found students, albeit in middle school, to nevertheless deploy irrelevant, personal associations (e.g., their uncle traveling to Italy, when learning about the fall of the Roman Empire) even when reasoning about a topic in history. This speaks to the domain general role of prior knowledge, of different types (e.g., domain, topic, and world knowledge and personal experience), in supporting (or impeding) comprehension across domains.
While we expected students’ general engagement of relevant prior knowledge to play a facilitative role in learning, in this study, we specifically selected a topic and texts that would speak both to students’ general world knowledge of schooling practices in the United States and to their own personal experiences as learners. We were unsure regarding the specific role such personal experiences would play in the effectiveness of students’ reasoning. We further thought that the controversial nature of this topic might further cause students to consider and reflect on their own opinions and points of view when interacting with the study texts. In this study, we were particularly interested in whether elements of contextualization (e.g., inclusion of prior knowledge or opinion) would manifest in the notes and written responses that students composed when learning based on multiple texts.
Task assignment
With these topic-related affordances in mind, in this study, we randomly assigned students to one of three different task conditions asking them to engage one of three heuristics during multiple text use (i.e., to engage in sourcing, corroboration, or contextualization). In accordance with the task assignments that students received, we examined students’ processing and writing performance for evidence of sourcing, corroboration, and contextualization. Prior work has found task assignment (e.g., asking students to write a summary or an argument) to be a powerful manipulation to improve students’ integration and writing based on multiple texts (Cerdán & Vidal-Abarca, 2008; Gil, Bråten, Vidal-Abarca, & Strømsø, 2010a, 2010b; Wiley & Voss, 1999), with task assignment serving a number of functions, including directing students’ attention to task-relevant information and helping students decide on which strategies to use (McCrudden & Schraw, 2007). Nevertheless, prior work manipulating task assignment within the context of students’ learning from multiple texts has done so by assigning students to produce different types of academic outcomes, with assignments focused on knowledge transforming (e.g., arguments) rather than knowledge telling (e.g., summaries), found to offer benefits for learning (List, Du, & Wang, 2019; Wiley & Voss, 1999). Nevertheless, in this study, in addition to asking students to produce one particular type of performance outcome (i.e., to compose a research report), we also explicitly directed students to engage in a particular type of strategic approach (i.e., sourcing, corroboration, or contextualization) during processing. We did this to direct students’ decision making regarding strategy use during task completion, suggested by Bohn-Gettler and McCrudden (2018) as a promising avenue for encouraging students’ more integrative processing when learning from multiple texts.
Thus we were interested in the extent to which students’ engagement in sourcing, corroboration, and contextualization increased learning from multiple texts. In this study, we assessed learning from multiple texts by examining students’ writing. As per the Documents Model Framework (Britt et al., 1999; Perfetti et al., 1999) and related empirical work in this area (e.g., Wiley & Voss, 1999), we expected better better quality writing to reflect (a) students’ inclusion of substantial information from the study texts, (b) their elaboration or transformation of this information, as a contrast to only directly paraphrasing text-based information, (c) their integration or thematic linking of information across texts, and (d) their attribution of information to source(s) of origin, allowing for verification. We expected students’ assignment to each of the three heuristic conditions examined in this study to specifically target at least one of these aspects of writing quality. For instance, contextualization was expected to support students’ elaboration or explanation of text-based information using prior knowledge. Corroboration was expected to increase students’ integration or thematic linking of information across texts, while sourcing was expected to increase students’ citation use or attribution of information to sources of origin. This was, in addition to, our more general expectation that students who were more engaged during multiple text use (e.g., accessing more texts, for longer periods of time, including more information in their notes) would also demonstrate better writing performance when composing a research report based on multiple texts (Bråten, Brante, & Strømsø, 2018).
We expected students’ engagement of all three heuristics (i.e., sourcing, corroboration, contextualization), jointly, to produce maximum benefits for students’ learning, as has been demonstrated in prior work (Rouet et al., 1997; Wineburg, 1991). Nevertheless, in this study, we directed students to engage in only one type of strategy use per condition. We did this for two related reasons. First, we wanted to analyze and compare the distinctive manifestation of each of these heuristics when students completed a multiple text task, beyond the domain of history. That is, we wanted to establish the extent to which all three of the strategies identified by Wineburg (1991) were comparably effective for students’ multiple text learning or whether some of these strategies seemed to have an outsized effect. Second, we wanted to determine whether brief, process-focused task instructions were sufficient to elicit variable strategic processing on the part of learners. Because of this second goal, we expected the introduction of all three strategies prior to reading to be overly demanding for the students in our sample. Indeed, when the use of all three of these heuristics has been examined in prior work, it has been investigated with relatively expert samples (e.g., professors, graduate students; e.g., Rouet et al., 1997; Wineburg, 1991), rather than with undergraduates, although this has not always been the case (e.g., Nokes et al., 2007).
We had the following research questions:
-
1
To what extent does students’ information use during a multiple text task differ in association with strategy assignment (i.e., asking students to engage in sourcing, corroboration, or contextualization)?
Given prior work on the effects of task assignments on subsequent processing (Gil et al., 2010a; McCrudden & Schraw, 2007; Wiley & Voss, 1999) we expected task assignments to change processing, in association with each condition. That is, we expected students in the sourcing condition to access more document information (e.g., author and publisher information) during reading and to include more sourcing information in their notes and more citations in their written responses. Similarly, we expected students in the corroboration condition to revisit texts as well as to include more cross-textual connections in their notes and written responses. We had less clear expectations regarding the processing of students assigned to the contextualization condition; nevertheless, we expected these students to include more information based on prior knowledge and experience in their notes and in the written responses that they composed.
-
2.
What is the nature of the association between students’ information use during multiple text processing and writing based on multiple texts (i.e., what is the association among the number of texts that students access, the volume of text-based information that students include in their notes, and the volume of text-based information students include in the written responses that they compose)?
Consistent with prior work examining the associations among students’ multiple text processing and writing quality (Anmarkrud et al., 2014; Du & List, 2020; Hagen et al., 2014), we expected all of these indicators to be associated with one another. In particular, we expected students accessing more texts to then include more text-based information in their notes and in the written responses that they composed. This is also consistent with Kobayashi’s (2009) work on the use of external strategies (e.g., note-taking) supporting students’ learning from multiple texts.
-
3.
To what extent do students’ strategy assignment and patterns of information use predict performance on a multiple text task?
Given prior work identifying the association between deep-level strategy use and task performance (Bråten, Anmarkrud, Brandmo, & Strømsø, 2014; Du & List, 2020; List & Alexander, 2020; List, 2020), we expected both task assignment and resulting strategy use (e.g., sourcing, corroboration) to be associated with improved writing based on multiple texts. We did not have a specific hypothesis regarding whether sourcing or corroboration would offer larger benefits for task performance; however, we did expect students assigned to the sourcing and corroboration conditions to outperform students assigned to the contextualization condition. While prior knowledge has been found to play a facilitative role in students’ learning from multiple texts (Barzilai & Strømsø, 2018; Gil et al., 2011a, 2010b; Le Bigot & Rouet, 2007), in this study, we did not specifically expect these prior knowledge-related benefits to outweigh those associated with instructing students to engage in deliberate multiple text processing-related strategy use (e.g., corroboration). This expectation was based on prior research which has found prior knowledge to have not only a facilitative effect for learning, but, at times, to be detrimental as well (Wolfe & Goldman, 2005).
With regard to strategic processing, we expected variables reflective of increased engagement with text-based information to be associated with improved task performance. Specifically, we expected that accessing more texts, for longer periods of time, and recording more information in notes would be associated with improvements in writing quality. We considered this to be the case both because increased information access was likely to facilitate students’ text-based response composition (e.g., allowing more text-based evidence to be included in writing, Du & List, 2020) and because we considered increased access to serve as an indicator of students’ effort, persistence, and deeper-level processing during reading (Bråten, Brante, & Strømsø, 2018; List, Stephens, & Alexander, 2019).
Methods
Participants
Participants were 72 undergraduate students from a large university in the Mid-Western United States (age: M = 19.97, SD = 1.27). Students were majority female (70.83%, n = 51; male: 23.61%, n = 17) and White (75.00%, n = 54). Two students (2.78%) reported their ethnicities as Black or African American. One student (1.39%) reported his or her ethnicity as Hispanic/Latino. Students came from a variety of education-related majors, including elementary education, special education, and speech-language pathology. Four students did not report any demographic information. Fifteen students did not report race/ethnicity.
Procedures
This study consisted of two primary parts. First, participants were asked to complete a variety of individual difference measures (i.e., prior knowledge, interest, and attitudes with regard to gifted education). Second, participants were asked to complete a multiple text task on the topic of gifted education. Specifically, this multiple text task included a research phase and a response phase, during which students were asked to use a library of six digital texts to research the topic of gifted education (i.e., research phase) before writing an argument on the issue (i.e., response phase). Students were assigned to one of three heuristic conditions prior to completing the multiple text task, or asked to engage in sourcing, corroboration, or contextualization.
Individual difference measures
Three individual difference measures were collected in this study (i.e., prior knowledge, interest, and students’ attitudes toward gifted education). While prior knowledge was used as a control in analyses, interest and attitudes are presented descriptively. Therefore, information about how interest and attitudes were assessed is included in Appendix 1. Neither interest (p = .79) nor any of the attitude-related measures (ps > .07) were found to significantly differ across task conditions.
Prior knowledge Prior knowledge was assessed via a term identification measure. Students were asked to provide their definitions for eight terms related to the topic of gifted and talented education. Each term was binary coded as correct or incorrect. Two researchers coded 28 student responses, reflecting 38.89% of participants. Cohen’s kappa inter-rater agreement was .89, with 94.64% exact agreement between raters. The internal consistency of prior knowledge items was Cronbach’s \(\alpha\) = .42; this increased to \(\alpha\) = .51 when one term was excluded.
Multiple text task
The multiple text task had three parts. First, students were assigned to a heuristic condition, or asked to engage in sourcing, corroboration, or contextualization. Then students were asked to research the topic of gifted education using a library of six digital texts as well as to compose a written response on the topic of gifted education. Specifically, students were asked to respond to the prompt: write an argument explaining whether or not your elementary school should have a gifted and talented program.
Experimental assignment Prior to starting the multiple text task, students were randomly assigned to one of the three experimental conditions, representing the three heuristics that Wineburg (1991) identified. Specifically, students in the sourcing condition were instructed: “In researching your argument, please consider where the information you are using comes from and who wrote it.” Students in the corroboration condition were asked to “compare the information sources to determine what they agree or disagree about” while researching. Finally, students in the contextualization condition were asked to “consider how this information relates to your prior knowledge and how it may apply in real-world school contexts.”
Research phase During the research phase, students were provided with a library of six digital texts to use in researching the target prompt, with three texts presenting arguments for gifted and talented education and three texts presenting arguments against it. All texts were intended to be trustworthy in nature (i.e., written in a professional style and attributed to reputable publishers). Texts were drawn from the Room for Debate segment from the New York Times and modified for inclusion in this study: https://www.nytimes.com/roomfordebate/2014/06/03/are-new-york-citys-gifted-classrooms-useful-or-harmful. Texts ranged from 249 to 256 words in length (M = 254.33. SD = 2.43). See Table 1 for information about the texts used in this study.
During the research phase of the study, log data of students’ text access were gathered and students were allowed to take notes on the texts provided. Table 2 includes a description of all indicators examined in this study.
Log data Log data included information on which and how many texts students accessed, whether or not students revisited texts, and the total time that students devoted to text access. Moreover, when students accessed texts in the digital library, only the title and content of a given text was loaded. Students had to access document information about each text (e.g., information about author and publisher) by clicking an additional button labeled: “click here to learn more about this source.” Including this button allowed us to further log whether or not students accessed document information in association with each text visited.
Students’ document information access was used as a metric of sourcing, while text revisits were intended as a metric of corroboration, corresponding to students’ iterative access of texts to compare these to one another (List & Alexander 2018). No log-based measures of contextualization were used.
Notes Students’ notes were analyzed using a four step process. First, notes were segmented into statements, typically including a single distinct idea that was off-set via a bullet or spacing. Then, statements were coded according to their content; including: (a) information about the task instructions, (b) source information (e.g., author, credentials), and (c) text-based content (i.e., number of texts included, number of statements drawn from each text). Inter-rater agreement for the number of statements from each text included in students’ notes was determined by two raters scoring 25 students’ notes (34.72%). Inter-rater agreement was 90.67% (n = 136), with disagreements revised through discussion.
Notes were also holistically examined for their inclusion of evaluations, cross-textual connections, and prior knowledge or opinions (see Table 2). Specifically, evaluations, along with the inclusion of sourcing information in students’ notes more generally, were intended to reflect sourcing. Cross-textual connection formation was taken to reflect corroboration. Finally, the inclusion of statements in notes reflecting prior knowledge or personal opinion was used as an indicator of contextualization. Holistic scoring was done by two raters for all students’ notes; exact agreement ranged from 100% (n = 60) for whether or not students included task instructions in their notes to 70% (n = 42) for the inclusion of prior knowledge. All disagreements were resolved through discussion.
As a final step in coding, information from students’ notes was compared to the information ultimately included in the written responses that students composed. This included considering (a) how many texts, of those accessed and included in students’ notes, were reflected in students’ written responses, (b) how many statements, from which texts, students included in the responses that they composed, and (c) whether the information included in students’ writing followed the same order as information in students’ notes, or whether backtracking (i.e., recursion or the inclusion of notes-based information out of sequence) was evidenced. This backtracking metric, in particular, was considered to reflect students’ more flexible, dynamic, and cross-textually focused information use. See Appendix 2 for a sample notes page.
Response phase Students were asked to write an argument in favor of or in opposition to their school adopting a gifted and talented program. Students’ responses were coded in two ways, analytically and for quality. While the analytic response coding classified each sentence included in students’ writing into one of ten possible categories, the quality score assigned to students’ writing reflected both the degree of argumentative reasoning that students’ writing exhibited and students’ overall writing performance.
Analytic response coding Students’ responses were coded analytically. That is, each sentence identified in students’ writing was first coded as text-based or not, and then further identified according to its purpose (e.g., stating students’ positions for or against gifted and talented education; linking information across two texts). Sentences in students’ writing were chosen as the focal units of analysis as these represented distinct segments of thought, as indicated by students themselves. A total of ten categories were identified, five corresponding to text-based sentences included in students’ written responses and five capturing sentences that were non-text-based. See Table 2 for definitions of each coding category and Table 3 for examples. See Appendix 3 for our coding of a sample student response. Two raters coded each sentence included in 45 students’ written responses (62.50% of the sample). Exact agreement for each sentence coded in target students’ responses was 81.15% (n = 451), out of a total of 556 sentences coded (Cohen’s \(\kappa\) = .79). Disagreements were resolved through discussion. As a final analytic metric, the number of unique citations, referring to any one of the six library texts by title or author, was further totaled.
Although a variety of analytic metrics, characterizing students’ written responses, were examined in this study, some of metrics were specifically analyzed for evidence of sourcing, corroboration, and contextualization. First, students’ citation use was used as an indicator of sourcing. Inter-textual connection formation was used as an indicator of corroboration. Finally, the number of statements reflecting prior knowledge and personal experience included were used as indicators of contextualization. See Table 2 for a mapping of response indicator to heuristic task conditions. See Table 3 for sample statements and responses.
Quality scores Consistent with prior work (Anmarkrud et al., 2014), students’ written responses were assigned a single score reflecting writing quality. A six point scale was used, reflecting (a) whether students generated a claim regarding the inclusion of gifted and talented education in their schools or not (i.e., assigned a 1); (b) whether students provided a single reason or multiple reasons to support this claim (i.e., assigned scores of 2 and 3, respectively); and (c) whether students acknowledged that there was a conflicting perspective, provided evidence consistent with this conflicting perspective, and refuted, reconciled, or otherwise balanced this conflicting perspective with their own point of view (i.e., assigned scores of 4, 5, and 6, respectively). As such, responses receiving a 6 on our rubric were both more elaborated in support of the inclusions of gifted and talented education in schools, or not, and were two-sided in the arguments these introduced. Scores were assigned in a hierarchical fashion, with higher quality responses including all of the features of their lower-level counterparts. For instance, if students made conflicting claims regarding the quality of gifted education, but did not provide any evidence in support of those claims, this was coded as a 1, rather than as a 4. That is, a score of 1 was applied because, in this example, no supporting evidence was provided, even though students did recognize a conflicting perspective introduced across texts. However, such responses were exceedingly rare. As demonstrated in the sample responses in Table 4, the vast majority of students’ responses were scored as at least a two (i.e., indicating that students were providing at least a single piece of evidence in support of a single claim). The only response scored using this decision rule was assigned a 1 and is included in Table 4. Cohen’s kappa inter-rater agreement was .66, indicating moderate agreement and corresponding to 71.83% exact agreement between raters. All disagreements were resolved through discussion.
Results
Because of the volume of data collected as a part of this study, results are presented in three parts. First, each data source (i.e., log data, students’ notes, and students’ written responses) is compared across conditions, when students were asked to engage in sourcing, corroboration, or contextualization. Next, descriptives of students’ multiple text use are presented, across conditions. Finally, a series of hierarchical regression models are run predicting response quality based on condition, log data of students’ text access, and information included in students’ notes. All variables analyzed and their correspondence to the three assigned heuristic conditions examined are presented in Table 2. Descriptive information is in Table 5 and information about skewness and kurtosis is in Table 6. Two variables of interest (i.e., the total number of text-based sentences and the number of inter-textual connections that students formed) were found to be non-normally distributed to a considerable extent. Analyses of these should be interpreted with caution, although the analytic procedures used (i.e., F-test) are robust to violations of normality (Blanca et al., 2017).
Research question 1. Multiple text use by condition
Our first research question examined the differences in multiple text use, as captured by log data, notes, and written responses, among students who were assigned to different heuristic conditions, or asked to engage in sourcing, corroboration, or contextualization.
Log data Among the log data indicators examined for this question, we expected document information access to be associated with students’ assignment to the sourcing condition and text revisits to be particularly common among students in the corroboration condition.
Overall One-way ANOVAs were run to examine differences in the number of texts visited and the total time devoted to text access among students assigned to different heuristic conditions. However, none of these results were significant, ps > .59.
Document information access Another one-way ANOVA was run to examine the differences in the number of texts for which students accessed document information in association with heuristic condition. The ANOVA was significant, F(2, 65) = 14.51, p < .001, \(\eta\) 2 = .31. Post-hoc analyses using Tukey’s HSD showed that students who were assigned to the sourcing condition accessed document information about significantly more texts (M = 3.70, SD = 2.38) than students assigned to the corroboration (M = 1.52, SD = 2.14) and contextualization (M = .58, SD = 1.47), ps < .01 conditions.
Revisits A Chi squared test of association was run to examine whether students revisiting texts during access was associated with heuristic condition. However, this association was not significant, p = .54.
All told, only the volume of document information access was found to differ across heuristic conditions and to be more frequent among students assigned to engage in sourcing while learning from multiple texts.
Notes With regard to the information included in students notes, we expected that students assigned to the sourcing condition would include both more document information and more frequent evaluations in their notes. We expected students in the corroboration condition to include more cross-textual connections in their notes and to evidence more backtracking. Finally, we expected students assigned to the contextualization condition to include more statements reflecting their prior knowledge or opinions in their notes.
Overall A Chi squared test of association was run to examine whether students’ inclusion of task instructions in their notes, or not, was associated with heuristic condition. However, this association was not significant, p = .81. Next we examined whether students’ notes across heuristic conditions differed in the amount of document information included, the number of texts featured, and the number of text-based statements recorded. A series of one-way ANOVAs were run. The volume of document information included in students’ notes differed by heuristic condition, F(2, 67) = 19.49, p < .001, \(\eta\) 2 = .37. Post-hoc analyses using Tukey’s HSD showed that students who were assigned to the sourcing condition included significantly more document information in their notes (M = 2.74, SD = 2.51), as compared to students assigned to the corroboration (M = .18, SD = .66) and contextualization (M = .28, SD = .89) conditions, ps < .001. However, across conditions, students’ notes did not differ in the amount of texts that they featured (p = .32) nor in the number of text-based statements in students’ notes (p = .97). Students’ engagement in backtracking was also not found to differ by heuristic condition (p = .38)
Evaluation, cross-textual connection formation, and prior knowledge/opinion inclusion A series of Chi squared tests were run to examine whether the inclusions of evaluations, cross-textual connections, and prior knowledge or opinion in students’ notes differed across heuristic conditions. However, these did not, ps > .58.
Only students’ inclusion of document information in their written notes was found to differ by heuristic condition.
Analytic written response scores A series of one-way ANOVAs were run to determine whether the analytic components of the responses that students composed differed in association with heuristic condition. Citation use was expected to be associated with students’ membership in the sourcing condition. We expected the number of inter-textual connections in students’ writing to be associated with their assignment to the corroboration task condition. Further, we expected students’ inclusion of prior knowledge and personal experiences in their written responses to be associated with assignment to the contextualization condition.
The number of text-based and non-text-based sentences, of various types, included in students’ written responses were not found to differ across task conditions, ps > .21. Still, the proportion of different types of sentences included in students' written responses, by condition, is presented in Figure 1. A Chi squared test of association was then run to examine whether the formation of cross-textual connections or not was associated with heuristic condition. However, this association was not found to be significant, p = .10. A one-way ANOVA was run to examine the differences in the number of citations included among students in different heuristic conditions. However, these differences were again not significant, p = .16.
Proportions of different types of sentences in students’ written responses across task conditions
Written response quality A final one-way ANOVA was run to determine whether students’ overall writing quality differed by task condition. However, no significant differences were found, p = .98.
No measures of writing quality were found to differ across task conditions.
Research question 2. Students’ process of multiple text use
Our second research question examined (a) log data of students’ text access, (b) students’ notes, and (c) written responses, composed in response to a multiple text task. Given the limited differences across heuristic conditions found in Research Question 1, analyses for Research Question 2 are collapsed across conditions. Descriptive information from these data sources is presented in Table 5. Information about students’ accessing and use of specific texts is included in Table 7. We further examined the associations among these. Due to concerns over length, only significant results are presented.
Associations among log data of students’ text access and information in students’ notes First, students were found to take notes on the majority of the texts that they accessed (81.94%, SD = .74). The number of text accessed was significantly associated with both the number of texts included in students’ notes, r(66) = .53, p < .001, and the total number of statements included, r(66) = .41, p < .001. As such, students accessing more information during multiple text use also included more of this information in their notes.
Second, including document information in notes was associated with the number of texts for which document information was accessed, r(66) = .71, p < .001. In particular, among students accessing any document information during navigation (45.83%, n = 33), an average of 52.19% (SD = .47) texts were then tagged with document information in students’ notes. That is, students accessing more document information during reading also included more of this information in their notes.
Associations among log data of text access and students’ analytic response scores The total number of texts that students accessed was significantly associated with the number of text-based sentences included in students’ written responses, r(68) = .30, p < .05. The number of citations included in students’ written responses was associated with log data of students’ document information access, r(68) = .29, p < .05. When students accessed more texts, this resulted in their including more text-based information in the written responses that they composed. Similarly, accessing more document information contributed to students’ citation use.
Associations among log data of text access and the quality of students’ written responses The total number of texts that students accessed was significantly associated with the quality of their written responses, r(68) = .30, p < .05. Accessing more texts was associated with better response quality.
Associations among notes and analytic written response scores Students’ written responses were further examined in association with the notes that students composed during the research phase of the study.
Document information in notes and written responses Students’ citation use was found to be significantly associated with their inclusion of source information in their notes, r(69) = .46, p < .001.
Texts reflected in students’ notes and written responses Overall, students included information from an average of 2.61 (SD = 1.36) texts in the written responses that they composed. The number of texts featured in students’ written responses, even if these were not explicitly cited, was significantly associated with the number of texts included in students’ notes, r(69) = .58, p < .001. In particular, students included an average of 70.45% (SD = .43) of the texts featured in their notes in the written responses that they composed. As such, students taking notes on more information then included this information in the written responses that they composed.
Content reflected in students’ notes and written responses Students included an average of 10.14 (SD = 8.02) text-based statements in their notes and an average of 4.59 (SD = 3.56) notes-based statements in their written responses. The number of text-based statements in students’ notes and notes-based statements in their written responses were significantly associated with one another, r(69) = .49, p < .001. This corresponded to students including an average of 54.70% of the information from their notes in the written responses that they composed (SD = .47). For us, this finding reflected students’ transference and culling of information from texts to their notes and from their notes to their writing.
Backtracking Backtracking is the non-linear use of information in notes. On average, students engaged in 1.11 (SD = 1.19) instances of backtracking during response composition. However, 38.89% (n = 28) of students only used their notes in a linear fashion. Students’ backtracking was significantly associated with the total number of text-based statement included in students’ notes, r(69) = .41, p < .001, as well as with the number of text-based statements, r(70) = .57, p < .001, and notes-based statements, r(70) = .73, p < .001, included in students’ written responses. Backtracking, as an indicator of more dynamic response composition, was associated with both the volume of information included in students’ notes as well as with the volume of text-based and notes-based information reflected in students’ writing.
Associations among students’ notes and the quality of students’ written responses Only the total number of texts included in students’ notes was associated with response quality, r(69) = .29, p < .05.
Associations among students’ analytic scores and written response quality Figure 2 shows the distribution of different types of statements in each student's written response. Students’ writing quality scores were significantly associated with the total number of sentences included in students’ responses [r(71) = .46, p < .001], the number of text-based sentences included [r(71) = .36, p < .01], the number of non-text-based sentences included [r(71) = .31, p < .01], as well as with the number of inter-textual connections formed during writing [r(71) = .27, p < .05]. The scores for writing quality assigned to students’ responses were significantly associated with the volume of information that students included (i.e., both text-based and non-text based) and with this information being integrated or linked across texts.
Representation of proportions of different types of sentences in each students’ written response
Research question 3. Predicting multiple text task performance
Our final research question examined the extent to which heuristic condition, log-data indicators of multiple text access, and metrics of text-based content in students’ notes could predict analytic written response components and overall writing quality. In particular, five multiple regression models were run, predicting the total number of statements, the number of text-based statements, the number of non-text-based statements, the number of cross-textual connections, and the number of citations included in students’ writing, respectively. Citation use and cross-textual connection formation were of particular interest to us in this study as these have been considered to be key indicators of successful multiple text integration in prior work (Britt & Aglinskas, 2002; List, Du, Wang, & Lee, 2019; Wiley & Voss, 1999). In this study, citation use was expected to be associated with students’ assignment to the sourcing condition. Cross-textual connection formation was expected to be associated with students’ membership in the corroboration task condition. And students’ inclusion of more non-text-based statements, overall, was expected to be predicted by their membership in the contextualization condition. A final, sixth, regression model was run predicting students’ overall writing quality.
For each regression model, prior knowledge was controlled for in Step 1, task condition was entered as a predictor in Step 2, with the sourcing condition coded as the referent group; log data indicators (i.e., total number of texts accessed, amount of document information accessed) were entered at Step 3; and metrics of students’ notes (i.e., number of statements in notes, volume of document information in notes, number of cross-textual connections, amount of back-tracking) were entered at Step 4. In selecting indicators, we were interested in choosing both those metrics that reflected the volume of information that students accessed and recorded (e.g., number of texts accessed; number of text-based statements in notes) and the quality of students’ information access and note-taking (e.g., document information access; cross-textual connections). Correlations among key variables are in Table 8.
After running each model, a residuals plot was examined to account for violations of normality and homoscedasticity. Three of the dependent variables examined (i.e., the number of inter-textual connections and citations in students’ written responses, students’ writing quality scores) were found to have non-homoscedastic residuals. To account for this, these three outcomes were predicted using weighed least squares regression. Multicollinearity statistics (i.e., tolerance and VIF) were inspected, with no problems identified (tolerance > .31, VIF < 3.20).
Total number of sentences A multiple regression model was run to predict the total number of sentences included in students’ written products. The final model was significant, F(9, 56) = 2.62, p < .05, \(R_{adj}^{2}\) = .18. Significant predictors in the model included the number of cross-textual connections (β = .25, p < .05) and backtracks (β = .33, p < .05) evidenced in students’ notes (See Table 9).
Total number of text-based statements A second regression model was run to predict the total number of text-based sentences (i.e., including text-direct statements, elaborations, transformations, intra-textual connections, and inter-textual connections) included in students’ writing. The overall model was significant, F(9, 56) = 6.21, p < .001, \(R_{adj}^{2}\) = .42. The number of texts accessed (β = .29, p < .05) as well as the volume of document information included in students’ notes (β = .36, p < .05), the number of cross-textual connections (β = .23, p < .05), and the number of backtracks (β = .53, p < .001) were all individually significant predictors in the model (See Table 10).
Total number of non-text-based statements A third multiple regression model was run to predict the total number of non-text based sentences featured in students’ written responses. However, the overall model was not significant, p = .73.
Number of cross-textual statements A fourth, weighted multiple regression model was run to predict the number of cross-textual connections that students included in the written responses that they composed. The overall model was significant, F(9, 56) = 4.37, p < .001, \(R_{adj}^{2}\) = .32. However, students’ backtracking through notes was the only individually significant predictor in the model (β = .61, p < .001) (See Table 11).
Citations A weighted multiple regression model was further run to predict the number of unique citations included in students’ written responses. The overall model was significant, F(9, 56) = 4.01, p < .001, \(R_{adj}^{2}\) = .29. Prior knowledge (β = .39, p < .05), the volume of document information included in students’ notes (β = .57, p < .001), and the number of back-tracks (β = .30, p < .05) were found to be individually significant predictors in the model (see Table 12).
Scores for writing quality A final, weighted multiple regression model was run to predict students’ writing quality scores. This model was significant, overall, F(9, 56) = 5.69, p < .001, \(R_{adj}^{2}\) = .39. The number of texts that students accessed (β = .38, p < .01) and the number of cross-textual connections included in students’ notes (β = .71, p < .001) were found to be individually significant predictors in the model (see Table 13).
Discussion
This study was carried out with both a primary and secondary purpose in mind. The primary purpose of this study was to determine whether three multiple text use strategies (i.e., sourcing, corroboration, and contextualization), commonly emphasized in the domain of history, would support students’ multiple text use in another social scientific domain (i.e., education). Drawing on prior work using task instructions as a means of inculcating strategy use among learners (Anmarkrud, McCrudden, Bråten, & Strømsø, 2013; Wiley & Voss, 1999), in this study, we asked students to engage in sourcing, corroboration, or contextualization, and examined the extent to which this was associated with differences in multiple text processing and task performance. In particular, we captured students’ processing both through log data of their multiple text access and through the notes that they recorded while researching the topic of gifted and talented education. We captured task performance by analyzing the written responses that students composed based on their accessing of multiple texts.
Our secondary goal in this study was to track students’ information use throughout the course of multiple text task completion. This included examining log data of students’ multiple text access, the notes that students recorded, and the information included in the written responses that they composed. In particular, we adopted an analytic approach, tracing each statement included in students’ written responses to either their prior knowledge or to their notes and to the texts they accessed in the research phase of the study. We unite these dual purposes by examining whether information use, as captured via log data, students’ notes, and written responses differed in association with assigned heuristic condition. In this discussion, we first overview the extent to which students’ strategy use and multiple text task performance differed when students were asked to engage in sourcing, corroboration, or contextualization. Then we consider, more generally, the nature of students’ information use throughout the course of multiple text task completion. We conclude with a discussion of the implications and limitations associated with this study.
Multiple text processing and performance across heuristic conditions
In this study, we prompted students’ use of three strategies found to characterize the nature of expert multiple text use in the domain of history (i.e., sourcing, corroboration, and contextualization; Wineburg, 1991). In particular, we were interested in whether the use of these strategies would facilitate the multiple text use of novices outside of the domain of history, in the field of education. We wanted to interrogate the assumption that strategies like comparing and contrasting information across texts (i.e., corroboration) and applying prior knowledge to understand information in texts (i.e., contextualization) would be facilitative of learning, beyond history. Findings from this study suggest that while prompting students to engage in sourcing, or the consideration of texts’ document information, like author or publisher, resulted in differences in multiple text processing, directing students to engage in corroboration or contextualization did not produce any noteworthy results. Indeed, students in the sourcing condition were found to access more document information during multiple text use and to include more document information in their notes than students assigned to the corroboration and contextualization heuristic conditions. However, no other differences in multiple text access or note-taking emerged. These results are consistent with Nokes et al. (2007) who found sourcing to be the strategy that high school students most commonly used, as compared to corroboration or contextualization, after receiving an intervention intended to foster historical reasoning. For instance, while 70% of students were found to engage in sourcing, only 7% of students used contextualization.
These sourcing-centered results can be explained in one of three primary ways. First, it may be the case that while sourcing is a strategy that can be engaged through the provision of directed task instructions, the other two heuristics (i.e., corroboration and contextualization), cannot be elicited only by asking students to use these during processing. Rather students may need more specific instruction in how to engage in corroboration and contextualization to be able to effectively use these during processing. Such instruction may need to be substantial as Nokes et al. (2007) found no improvement in high-school students’ use of sourcing, corroboration, and contextualization after completing a 15-lesson intervention. At the same time, some prior work has found promise in developing interventions for sourcing, as a principal strategy for improving students’ learning from multiple texts (e.g., Britt & Aglinskas, 2002; Branten & Strømsø, 2009; Pérez et al., 2018; Stadtler & Bromme, 2007).
Second, sourcing may be more overt as a strategy, than corroboration or contextualization. In this study, there were a number of indicators readily available to capture the nature of students’ sourcing during both processing (i.e., accessing document information, recording document information in notes) and performance (i.e., citation use). But such indicators were less plentiful and less effective in capturing students’ engagement in corroboration and contextualization. For instance, in this study, we considered students’ revisiting of texts during the research phase of the study to reflect corroboration. While this interpretation is consistent with prior work (Goldman et al., 2012; List et al., 2019; List & Alexander, 2017; Wiley et al., 2009), students could certainly have revisited texts without engaging in corroboration or could have corroborated information across texts, using only their notes, without revisiting any of the texts they accessed. Likewise, we examined the inclusion of prior knowledge or personal opinion in students’ notes as an indicator of contextualization; at the same time, students may have brought their prior knowledge to bear on the information they accessed (i.e., engaging in contextualization), without necessarily recording any of this processing in their notes. As such, it is important to recognize that all three of the strategies examined in this study are primarily cognitive in nature, making capturing these during the course of multiple text processing and performance a methodological challenge, potentially reducing the associations between heuristic condition and processing indicators identified in this study.
Finally, it may also be the case that while sourcing is a strategy that is more universally applied across domains, corroboration and contextualization are strategies that function in a more domain-specific fashion. Certainly, across domains, there is a need to corroborate or compare and contrast information across texts. At the same time, the corroboration of historical accounts may be somewhat distinct from juxtaposing arguments for and against a particular issue, that are marshalling diverse evidence in support of conflicting claims. Likewise, contextualization, or the application of topic, domain, and world knowledge to understanding information presented across texts (Rouet et al., 1997), may function differently in the history domain vis-à-vis in domains where more general, world knowledge or personal experience may apply. Additional research is also needed to identify those strategies, beyond sourcing, corroboration, and contextualization, that may be central to effective learning from multiple texts in non-history domains. A key limitation of this study was that it did not include a control group. Therefore, we are unable to determine whether students assigned to different strategy conditions performed better than students would have otherwise (i.e., if not prompted to engage in strategy use at all). Rather, we can only compare the relative effectiveness of instructions to engage in sourcing, corroboration, or contextualization. Further, it may be the case that we did not see differences across strategy conditions because students already use these strategies to a considerable extent (i.e., and do not need further prompting to do so). We nevertheless consider this latter explanation to be unlikely given the limited evidence of sourcing, corroboration, and contextualization demonstrated in students’ notes and other measures of multiple text processing and considering prior work (e.g., Anmarkrud et al., 2014; List & Alexander, 2020).
Despite sourcing being associated with differences in students’ processing of multiple texts, to the exclusion of corroboration and contextualization, the limitations of this strategy for multiple text task performance ought be recognized as well. That is, while sourcing was associated with differences in students’ multiple text access and processing (e.g., document information accessed; document information included in students’ notes), it was not ultimately associated with differences in multiple text task performance, as captured via both analytic and quality-related written response scores. This may be the case for a variety of reasons. First, unlike prior work which has used task instructions as a means of directing students to produce some type of outcome (e.g., an argument or a research report, Wiley & Voss, 1999), in this study we asked students to engage in a mode of processing (i.e., to engage in sourcing, corroboration, or contextualization) during multiple text use. This focus on processing may have carried with it some limitations. That is, unlike simply assigning students to a task condition, for which students may have a robust schema (e.g., students know what writing a summary entails, List et al., 2019), students may not have had the declarative, procedural, and conditional knowledge necessary to effectively engage any of the strategies that they were assigned to use (Paris, Lipson, & Wixson, 1983). Indeed, extensive work on strategy instruction-related interventions suggests that task instructions alone may not be sufficient to engage students in integrative processing when learning from multiple texts (Macedo-Rouet, Braasch, Britt, & Rouet, 2013; Martínez, Mateos, Martín, Rijlaarsdam, 2015). In this study, only brief prompts were used to elicit students’ strategy use during reading. Although these were intentionally written in a manner that would be actionable for students, these brief prompts may not have been sufficiently elaborated to change the nature of students’ processing. Future work should consider using richer strategy inductions to improve students’ processing during multiple text task completion. Certainly, this is suggested by results from regression analysis and should be explored in further work. That is, students instructed to engage in sourcing did attend more to document information during processing. At the same time, assignment to the sourcing condition did not predict analytic writing performance, while the accessing of document information did, after heuristic condition was controlled for. This suggests that strategy assignment may result in differential strategy use (i.e., attendance to document information during processing), with that strategy use then manifesting in differences in task performance (i.e., the number of text-based statements and citations included in students’ responses).
Third, it may be that directions intended to foster any type of strategy use during multiple text access only produce a superficial, rather than deep-level, type of engagement. That is, instructions asking students to engage in sourcing did result in students accessing and making note of document information during text access but was not associated with indicators of evaluation in students’ notes nor with cross-textual connection formation or citation use in the written responses that students composed. This may mean that while sourcing instructions do prompt students’ attendance to document information, more metacognitive volition or deliberate strategic engagement is needed for students to engage in deeper level multiple text evaluation and integration. More generally, the nature of students’ writing based on multiple texts may be such a complex and multidimensional process that it requires a variety of strategies, beyond sourcing, to improve (Anmarkrud et al., 2014; Mateos, Solé, Martin, Cuevas, Miras, & Castells, 2014; Spivey, 1990). To the extent that providing task instructions to elicit the whole range of strategies that students may need to compose a quality written response is not a feasible option, more research is needed on how to encourage strategic writing on the part of learners. For insight into the types of strategies that may be required, we next examine the nature of students’ information use throughout the course of writing based on multiple texts.
Information use throughout text access, note-taking, and response composition
In addition to examining processing across heuristic conditions, we were further interested in examining students’ information use, more broadly, through the course of text access, note-taking, and written response composition. In the broader literature on learning from multiple texts, a variety of processes associated with information use have been identified (List & Alexander, 2019; Rouet & Britt, 2011). These include information selection, processing, evaluation, and integration. These processes were likewise manifest in this study, to varying extents. In particular, students’ engagement in two key processes (i.e., selection and integration) were notable in this study, one for its relative prevalence and the other for its notable absence.
Analytically tracking students’ information use throughout text access, note taking, and response composition revealed that these were primarily guided by a process of information selection or filtration. We use the term information selection here to refer to students’ deliberate decisions regarding which information in texts is important or relevant for task completion and which information is not. Such selection includes both the cognitive appraisal of information as relevant or not and the behavioral act of note-taking, to record that information that is deemed to be relevant or important. In this study, students selected a subset of information from texts to include in their notes and further filtered that information to determine which content to ultimately include in their written responses. Indeed, while metrics of text access, information included in students’ notes, and text-based information included in students’ written responses were all associated with one another, as demonstrated in Research Question 2, it was also the case that there was less information included in students’ notes than was available in the texts provided, and less text-based information in students’ written responses than was included in their notes. As such, multiple text processing and written response composition can be thought of as tasks requiring the progressive reducing of information (see Table 7). Although it is difficult to determine how much information is sufficient for students to include in their notes or in the written responses that they compose, students did seem to be at least adept in selecting among the volume of text-based information available.
At the same time, while selection seemed to be a robust and well-developed competency for learners, integration was found to be considerably less so. A variety of indicators, capturing students’ cross-textual connection formation during both multiple text processing and response composition were examined in this study. These included students revisiting texts during multiple text access, formulating cross-textual connections in their notes, and including instances of integration in the written responses that they composed. Across these various indicators, students’ cross-textual connection formation, associated with corroboration, was found to be quite limited in nature, despite its association with students’ quality of writing performance. Similar findings have been found in prior work documenting the challenges that students experience with cross-textual integration (Anmarkrud et al., 2014; Barzilai, Zohar, & Mor-Hagani, 2018; List, Du, Wang, & Lee, 2019; Wiley & Voss, 1999). This study both provides further evidence of such challenges and suggests that they cannot be ameliorated with task instructions alone (e.g., asking students to engage in corroboration). As such, developing more robust interventions to foster cross-textual integration, in addition to other strategies related to learning from multiple texts, represents an important direction for future work (Barzilai et al., 2018).
Further, more work is needed to consider which specific processing strategies may be most associated with students’ performance under various task conditions. In this study, students were asked to write an argument based on multiple texts. The outcomes measures used in this study pertained most to the multiple texts aspect of this assignment. That is, by analytically coding each sentence included in students’ writing, we specifically focused on students’ use of text-based information, their transformation of this information or its elaboration using prior knowledge, and, most of all, on students’ cross-textual information integration. In other words, the dependent variables examined in this study carried with them the tacit assumption that both more information (i.e., more sentences) and more text-based and cross-textual information, in particular, were reflective of better response quality. Certainly this is consistent with models of learning from multiple texts which identify both the comprehensive coverage of key issues about complex topics and the integrative understanding of these issues, as they are described across texts, to be key facets of effective learning from multiple texts (Britt et al., 1999; Du & List, 2020; List, Du, Wang, & Lee, 2019; Perfetti et al., 1999). This is further consistent with work suggesting that, in part, students’ provision of evidence (e.g., the inclusion of more text based information) in their writing is associated with better argument writing performance (Iordanou & Constantinou, 2014; Zembal-Saul, Munford, Crawford, Friedrichsen, & Land, 2002). Indeed, in the present study students’ writing quality scores were predicted both by the number of texts that students initially accessed (i.e., information volume) and the number of cross-textual connections that students included in their notes. Nevertheless, additional aspects of argument writing (e.g., forming a claim, developing counter-arguments) were not examined as reflected in students’ multiple text access or note-taking. As such, considering which argument-construction related processing variables are associated with writing performance, and, indeed, what these variables even may be, should doubtlessly be considered in future research.
Limitations
In addition to these major discussion points, a number of limitations associated with the study must also be acknowledged. First, the fundamental design of this study may have limited its applicability to the real world. While students in this study were deliberately assigned to engage in only one strategic approach when learning from multiple texts, in reality sourcing, corroboration, contextualization, alongside other strategy use, are all needed to foster students’ learning from multiple texts. As such, students’ assignment to only a singular heuristic task condition presents a major limitation in this study. Moreover, inherent in definitions of strategic processing and strategy use is the notion that such processing should be deliberate, goal-directed, and metacognitively directed on the part of learners (Paris et al., 1983). It remains unclear the extent to which assigning students to adopt a particular type of heuristic processing during task completion adequately simulates experts’ use of sourcing, corroboration, or contextualization, when they decide to engage such strategies during processing. This methodological limitation is compounded by the fact that we did not include a control group in this study, thereby limiting our understanding of what students may do by default when presented with a multiple text task, such as the one used in this study. Moreover, the specific task instructions that students received may not have been sufficient to elicit the depth of strategy engagement that we intended. For instance, directing students to engage in sourcing may have resulted in them only attending to source, rather than in considering source in their interpretations of information. Likewise, asking students to engage their prior knowledge and experiences, in the contextualization condition, may have over-directed students to rely on personal experience, rather than directing students to contextualize information using their academic knowledge of effective educational practice.
Second, the sample size used in this study was smaller than was likely needed. For instance, G*Power suggests that to conduct an ANOVA with three groups, and with \(\alpha\) = .05 and \(\beta\) = .80, by convention, a sample of 159 students is needed to identify a medium effect (f = .25). Under these same conditions (\(\alpha\) = .05, \(\beta\) = .80), we were only able to identify large effects (f > .40) for Research Question 1, with identifying these only requiring a sample of 66 students or more. Likewise, for a multiple linear regression, with 9 predictors (\(\alpha\) = .05, \(\beta\) = .80), a sample size of 114 students would be been needed to be able to identify medium-sized effects (f2 = .15, R2 > .13). At the same time, we were successfully able to identify large effects, for Research Question 3, for which only a sample of 54 students, or more, was needed (f2 = .35, R2 > .26).
Third, a methodological challenge arising in this study was associated with our desire to assess students’ sourcing, corroboration, and contextualization, with these ultimately constituting cognitive strategies, unable to be directly observed. The fundamentally cognitive nature of these heuristics may have resulted in the limited associations between task condition and indicators of strategy use identified in this study. Nevertheless, the future use of data-intensive methods, including eye-tracking (e.g., to trace students’ gaze-paths across texts) and think-alouds, to supplement analyses carried out in this study, may prove to be fruitful ways of capturing sourcing, corroboration, and contextualization. We can also draw on other research paradigms to better explore students’ strategy use during processing. For instance, some studies have used experimental designs to introduce students to foregrounding information (or not) thereby manipulating the potential for contextualization (Kurby, Britt, & Magliano, 2005; Maier & Richter, 2013). Students can also be asked to report their prior knowledge activation or engagement in corroboration, each time this occurs, similar to how Stadtler and Bromme (2007) asked students to rate the quality of their comprehension during reading. Still, investigating the extent to which students engage prior knowledge, of various types, when learning from multiple texts, and in sourcing and corroboration more broadly, remain as directions for future work.
Conclusion
In this study we sought to evaluate the effectiveness of three multiple text use strategies (i.e., sourcing, corroboration, and contextualization), previously identified based on expert studies in history (Wineburg, 1991). We sought to elicit the use of these strategies by providing students with process-focused task instructions, asking them to engage in sourcing, corroboration, or contextualization. While we found evidence for students’ use of sourcing during multiple text access and as reflected in students’ notes, use of corroboration and contextualization seemed to be more limited. At the same time, we found evidence for students’ progressively more filtered information selection as students moved from gathering information during reading, to recording text-based information in their notes, to including this information in the written responses that they composed. Moreover, we found some students to engage in integration and cross-textual relation formation, albeit to a limited extent. This was reflected in students drawing connections among texts in their notes, backtracking through notes or using notes dynamically during writing, and formulating integrative statements in the written responses that they composed. Although such integrative strategy use was not found to be facilitated by any of the three heuristic conditions to which students were assigned, this type of processing represents a key target for future intervention.
References
Anmarkrud, Ø., Bråten, I., & Strømsø, H. I. (2014). Multiple-documents literacy: Strategic processing, source awareness, and argumentation when reading multiple conflicting documents. Learning and Individual Differences, 30, 64–76. https://doi.org/10.1016/j.lindif.2013.01.007.
Anmarkrud, Ø., McCrudden, M. T., Bråten, I., & Strømsø, H. I. (2013). Task-oriented reading of multiple documents: Online comprehension processes and offline products. Instructional Science: An International Journal of the Learning Sciences, 41, 873–894. https://doi.org/10.1007/s11251-013-9263-8.
Barzilai, S., & Strømsø, H. I. (2018). Individual differences in multiple document comprehension. In J. L. G. Braasach, I. Bråten, & M. T. McCrudden (Eds.), Handbook of multiple source use (pp. 99–116). New York: Routledge.
Barzilai, S., Thomm, E., & Shlomi-Elooz, T. (2020). Dealing with disagreement: The roles of topic familiarity and disagreement explanation in evaluation of conflicting expert claims and sources. Learning and Instruction, 69, Advanced online publication. https://doi.org/10.1016/j.learninstruc.2020.101367
Barzilai, S., Zohar, A. R., & Mor-Hagani, S. (2018). Promoting integration of multiple texts: a review of instructional approaches and practices. Educational Psychology Review, 30, 973–999. https://doi.org/10.1007/s10648-018-9436-8.
Bigot, L. L., & Rouet, J. F. (2007). The impact of presentation format, task assignment, and prior knowledge on students’ comprehension of multiple online documents. Journal of Literacy Research, 39, 445–470. https://doi.org/10.1080/10862960701675317.
Blanca, M. J., Alarcón, R., Arnau, J., Bono, R., & Bendayan, R. (2017). Non-normal data: Is ANOVA still a valid option? Psicothema, 29(4), 552–557.
Bohn-Gettler, C. M., & McCrudden, M. T. (2018). Effects of task relevance instructions and topic beliefs on reading processes and memory. Discourse Processes, 55, 410–431. https://doi.org/10.1080/0163853X.2017.1292824.
Brante, E. W., & Strømsø, H. I. (2018). Sourcing in text comprehension: A review of interventions targeting sourcing skills. Educational Psychology Review, 30, 773–799. https://doi.org/10.1007/s10648-017-9421-7.
Bråten, I., Anmarkrud, Ø., Brandmo, C., & Strømsø, H. I. (2014). Developing and testing a model of direct and indirect relationships between individual differences, processing, and multiple-text comprehension. Learning and Instruction, 30, 9–24. https://doi.org/10.1016/j.learninstruc.2013.11.002.
Bråten, I., Brante, E. W., & Strømsø, H. I. (2018). What really matters: The role of behavioural engagement in multiple document literacy tasks. Journal of Research in Reading, 41, 680–699. https://doi.org/10.1111/1467-9817.12247.
Bråten, I., & Strømsø, H. I. (2009). Effects of task instruction and personal epistemology on the understanding of multiple texts about climate change. Discourse Processes: A Multidisciplinary Journal, 47, 1–31. https://doi.org/10.1080/01638530902959646.
Bråten, I., & Strømsø, H. I. (2011). Measuring strategic processing when students read multiple texts. Metacognition and Learning, 6, 111–130. https://doi.org/10.1007/s11409-011-9075-7.
Bråten, I., Strømsø, H. I., & Britt, M. A. (2009). Trust matters: Examining the role of source evaluation in students’ construction of meaning within and across multiple texts. Reading Research Quarterly, 44, 6–28. https://doi.org/10.1598/RRQ.44.1.1.
Bråten, I., Strømsø, H. I., & Salmerón, L. (2011). Trust and mistrust when students read multiple information sources about climate change. Learning and Instruction, 21, 180–192. https://doi.org/10.1016/j.learninstruc.2010.02.002.
Britt, M. A., & Aglinskas, C. (2002). Improving students’ ability to identify and use source information. Cognition and Instruction, 20, 485–522. https://doi.org/10.1207/S1532690XCI2004_2.
Britt, M. A., Perfetti, C. A., Sandak, R., & Rouet, J. F. (1999). Content integration and source separation in learning from multiple texts. In S. R. Goldman, A. C. Graesser, & P. van den Broek (Eds.), Narrative comprehension, causality, and coherence: Essays in honor of Tom Trabasso (pp. 209–233). Mahwah: Lawrence Erlbaum Associates.
Britt, M. A., & Rouet, J. F. (2012). Learning with multiple documents: Component skills and their acquisition. In J. R. Kirby & M. J. Lawson (Eds.), Enhancing the quality of learning: Dispositions, instruction, and learning processes (pp. 276–314). New York: Cambridge University Press.
Britt, M. A., Rouet, J. F., Blaum, D., & Millis, K. (2019). A reasoned approach to dealing with fake news. Policy Insights from the Behavioral and Brain Sciences, 6(1), 94–101. https://doi.org/10.1177/2372732218814855.
Burton, V. T., & Chadwick, S. A. (2000). Investigating the practices of student researchers: Patterns of use and criteria for use of Internet and library sources. Computers and Composition: An International Journal, 17, 309–328. https://doi.org/10.1016/S8755-4615(00)00037-2.
Cerdán, R., & Vidal-Abarca, E. (2008). The effects of tasks on integrating information from multiple documents. Journal of Educational Psychology, 100, 209–222. https://doi.org/10.1037/0022-0663.100.1.209.
Du, H., & List, A. (2020). Researching and writing based on multiple texts. Learning and Instruction, 66.
Fives, H., & Buehl, M. M. (2012). Spring cleaning for the “messy” construct of teachers’ beliefs: What are they? Which have been examined? What can they tell us?. In K. R. Harris, S. Graham, T. Urdan, S. Graham, J. M. Royer, & M. Zeidner (Eds.), Individual differences and cultural and contextual factors (Vol. 2, pp. 471–499)., APA Handbooks in Psychology®. APA educational psychology handbook Washington, D.C: American Psychological Association.
Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010a). Summary versus argument tasks when working with multiple documents: Which is better for whom? Contemporary Educational Psychology, 35, 157–173. https://doi.org/10.1016/j.cedpsych.2009.11.002.
Gil, L., Bråten, I., Vidal-Abarca, E., & Strømsø, H. I. (2010b). Understanding and integrating multiple science texts: Summary tasks are sometimes better than argument tasks. Reading Psychology, 31, 30–68. https://doi.org/10.1080/02702710902733600.
Goldman, S. R. (2004). Cognitive aspects of constructing meaning through and across multiple texts. In N. Shuart-Faris & D. Bloome (Eds.), Uses of intertextuality in classroom and educational research (pp. 317–352). Greenwich: Information Age.
Goldman, S. R., Braasch, J. L., Wiley, J., Graesser, A. C., & Brodowinska, K. (2012). Comprehending and learning from Internet sources: Processing patterns of better and poorer learners. Reading Research Quarterly, 47, 356–381. https://doi.org/10.1002/RRQ.027.
Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., et al. (2016). Disciplinary literacies and learning to read forunderstanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219–246.
Gottschling, S., Kammerer, Y., & Gerjets, P. (2019). Readers’ processing and use of source information as a function of its usefulness to explain conflicting scientific claims. Discourse Processes, 56, 429–446. https://doi.org/10.1080/0163853X.2019.1610305.
Hagen, Å. M., Braasch, J. L., & Bråten, I. (2014). Relationships between spontaneous note-taking, self-reported strategies and comprehension when reading multiple texts in different task conditions. Journal of Research in Reading, 37(S1), S141–S157. https://doi.org/10.1111/j.1467-9817.2012.01536.x.
Hartman, D. K., & Hartman, J. A. (1993). Reading across texts: Expanding the role of the reader. The Reading Teacher, 47, 202–211.
Havlicek, L. L., & Peterson, N. L. (1976). Robustness of the Pearson correlation against violations of assumptions. Perceptual and Motor Skills, 43(3_suppl), 1319–1334. https://doi.org/10.2466/pms.1976.43.3f.1319
Hyland, K., & Bondi, M. (Eds.). (2006). Academic discourse across disciplines (Vol. 42). New York: Peter Lang.
Iordanou, K., & Constantinou, C. P. (2014). Developing pre-service teachers’ evidence-based argumentation skills on socio-scientific issues. Learning and Instruction, 34, 42–57. https://doi.org/10.1016/j.learninstruc.2014.07.004.
Kendeou, P., Robinson, D. H., & McCrudden, M. T. (2019). Misinformation and disinformation in education: An introduction. In P. Kendeou, D. H. Robinson, & M. T. McCrudden (Eds.), Misinformation and fake news in education (pp. 1–6). Charlotte: Information Age.
Kobayashi, K. (2009). Comprehension of relations among controversial texts: Effects of external strategy use. Instructional Science, 37, 311–324. https://doi.org/10.1007/s11251-007-9041-6.
Kurby, C. A., Britt, M. A., & Magliano, J. P. (2005). The role of top-down and bottom-up processes in between-text integration. Reading Psychology, 26, 335–362. https://doi.org/10.1080/02702710500285870.
List, A. (2020). Six questions regarding strategy use when learning from multiple texts. In D. L. Dinsmore, L. Fryer, & M. M. Parkinson (Eds.), Handbookof strategies and strategic processing: conceptualization, intervention, measurement, and analysis (pp. 119–140). New York: Routledge.
List, A., & Alexander, P. A. (2017). Text navigation in multiple source use. Computers in Human Behavior, 75, 364–375.
List, A., & Alexander, P. A. (2018). Corroborating students’ self-reports of source evaluation. Behaviour & Information Technology, 37(3), 198–216.
List, A., & Alexander, P. A. (2019). Toward an integrated framework of multiple text use. Educational Psychologist, 54(1), 20–39.
List, A., & Alexander, P. A. (2020). Strategy use in learning from multiple texts: An investigation of the Integrative Framework of Learning from multipletexts. Frontiers in Education, 5.
List, A., Alexander, P. A., & Stephens, L. A. (2017). Trust but verify: Examining the association between students’ sourcing behaviors and ratings of texttrustworthiness. Discourse Processes, 54(2), 83–104.
List, A., Du, H., Wang, Y. (2019). Understanding students’ conceptions of task assignments. Contemporary Educational Psychology, 59, 101801.
List, A., Du, H., Wang, Y., & Lee, H. Y. (2019). Toward a typology of integration: Examining the documents model framework. ContemporaryEducational Psychology, 58, 228–242.
List, A., & Rubenstein, L. D. (2019). Understanding susceptibility to educational inaccuracies: Examining the Likelihood of Adoption Model. In P. Kendeou, D. Robinson, & M. T. McCrudden (Eds.), Misinformation and Fake News in Education (pp. 29–53). Charlotte, NC: Information Age Publishing.
List, A., Stephens, L. A., & Alexander, P. A. (2019). Examining interest throughout multiple text use. Reading and Writing, 32(2), 307–333.
Macedo-Rouet, M., Braasch, J. L., Britt, M. A., & Rouet, J. F. (2013). Teaching fourth and fifth graders to evaluate information sources during text comprehension. Cognition and Instruction, 31, 204–226. https://doi.org/10.1080/07370008.2013.769995.
Maier, J., & Richter, T. (2013). Text belief consistency effects in the comprehension of multiple texts with conflicting information. Cognition and Instruction, 31, 151–175. https://doi.org/10.1080/07370008.2013.769997.
Many, J. E. (1996). Patterns of selectivity in drawing on sources: Examining students’ use of intertextuality across literacy events. Literacy Research and Instruction, 36, 51–63. https://doi.org/10.1080/19388079609558227.
Martínez, I., Mateos Sanz, M. D. M., Martín, E., & Rijlaarsdam, G. (2015). Learning history by composing synthesis texts: Effects of an instructional programme on learning, reading and writing processes, and text quality. Journal of Writing Research., 7, 275–302. https://doi.org/10.17239/jowr-2015.07.02.03.
Mateos, M., Solé, I., Martín, E., Cuevas, I., Miras, M., & Castells, N. (2014). Writing a synthesis from multiple sources as a learning activity. In P. Klein, P. Boscolo, L. Kirkpatrick, & C. Gelati (Eds.), Writing as a learning activity (pp. 169–190). Leiden: Brill.
McCrudden, M. T., & Schraw, G. (2007). Relevance and goal-focusing in text processing. Educational Psychology Review, 19, 113–139. https://doi.org/10.1007/s10648-006-9010-7.
McCarthy, K. S., & Goldman, S. R. (2019). Constructing interpretive inferences about literary text: The role of domain-specific knowledge. Learning andInstruction, 60, 245–251.
Nokes, J. D., Dole, J. A., & Hacker, D. J. (2007). Teaching high school students to use heuristics while reading historical texts. Journal of Educational Psychology, 99, 492–504. https://doi.org/10.1037/0022-0663.99.3.492.
Paris, S. G., Lipson, M. Y., & Wixson, K. K. (1983). Becoming a strategic reader. Contemporary Educational Psychology, 8, 293–316. https://doi.org/10.1016/0361-476X(83)90018-8.
Pérez, A., Potocki, A., Stadtler, M., Macedo-Rouet, M., Paul, J., Salmerón, L., et al. (2018). Fostering teenagers’ assessment of information reliability: Effects of a classroom intervention focused on critical source dimensions. Learning and Instruction, 58, 53–64. https://doi.org/10.1016/j.learninstruc.2018.04.006.
Perfetti, C. A., Rouet, J.-F., & Britt, M. A. (1999). Towards a theory of documents representation. In H. van Oostendorp & S. R. Goldman (Eds.), The Construction of mental representations during reading (pp. 99–122). Mahwah: Lawrence Erlbaum Associates.
Reisman, A. (2012). Reading like a historian: A document-based history curriculum intervention in urban high schools. Cognition and instruction, 30(1), 86–112. https://doi.org/10.1080/07370008.2011.634081.
Reisman, A., & Wineburg, S. (2008). Teaching the skill of contextualizing in history. The Social Studies, 99, 202–207. https://doi.org/10.3200/TSSS.99.5.202-207.
Richter, T. (2011). Cognitive flexibility and epistemic validation in learning from multiple texts. In J. Elen, E. Stahl, R. Bromme, & G. Clarebout (Eds.), Links between beliefs and cognitive flexibility. (pp. 125–140). Dordrecht: Springer. https://doi.org/10.1007/978-94-007-1793-0_7.
Richter, T., & Maier, J. (2017). Comprehension of multiple documents with conflicting information: A two-step model of validation. Educational Psychologist, 52(3), 148–166. https://doi.org/10.1080/00461520.2017.1322968.
Ronan, B. (2015). Intertextuality and dialogic interaction in students’ online text construction. Literacy Research: Theory, Method, and Practice, 64, 379–397. https://doi.org/10.1177/2381336915617613.
Rouet, J. F., & Britt, M. A. (2011). Relevance processes in multiple docu- ment comprehension. In M. T. McCrudden, J. P. Magliano, & G. Schraw (Eds.), Relevance instructions and goal-focusing in text learning (pp. 19–52). Greenwich: Information Age.
Rouet, J. F., Favart, M., Britt, M. A., & Perfetti, C. A. (1997). Studying and using multiple documents in history: Effects of discipline expertise. Cognition and Instruction, 15, 85–106. https://doi.org/10.1207/s1532690xci1501_3.
Salmerón, L., Gil, L., & Bråten, I. (2018). Using eye-tracking to assess sourcing during multiple document reading: A critical analysis. Frontline Learning Research, 6(3), 105–122.
Schmidt, A. F., & Finan, C. (2018). Linear regression and the normality assumption. Journal of clinical epidemiology, 98, 146–151. https://doi.org/10.1016/j.jclinepi.2017.12.006
Shanahan, C. (2009). Disciplinary comprehension. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 240–260). New York: Routledge.
Smagorinsky, P., & O’Donnell-Allen, C. (1998). Reading as mediated and mediating action: Composing meaning for literature through multimedia interpretive texts. Reading Research Quarterly, 33, 198–226. https://doi.org/10.1598/RRQ.33.2.3.
Spivey, N. N. (1990). Transforming texts: Constructive processes in reading and writing. Written Communication, 7, 256–287. https://doi.org/10.1177/0741088390007002004.
Stadtler, M., & Bromme, R. (2007). Dealing with multiple documents on the WWW: The role of metacognition in the formation of documents models. International Journal of Computer-Supported Collaborative Learning, 2, 191–210. https://doi.org/10.1007/s11412-007-9015-3.
Stadtler, M., & Bromme, R. (2013). Multiple document comprehension: An approach to public understanding of science. Cognition and Instruction, 31, 122–129. https://doi.org/10.1080/07370008.2013.771106.
Stadtler, M., & Bromme, R. (2014). The content-source integration model: A taxonomic description of how readers comprehend conflicting scientific information. In D. N. Rapp & J. L. G. Braasch (Eds.), Processing inaccurate information: Theoretical and applied perspectives from cognitive science and the educational sciences (pp. 379–402). Cambridge: MIT Press.
Stahl, S. A., Hynd, C. R., Britton, B. K., McNish, M. M., & Bosquet, D. (1996). What happens when students read multiple source documents in history? Reading Research Quarterly, 31, 430–456. https://doi.org/10.1598/RRQ.31.4.5.
Strømsø, H. I., Bråten, I., & Samuelstuen, M. S. (2003). Students’ strategic use of multiple sources during expository text reading: A longitudinal think-aloud study. Cognition and Instruction, 21, 113–147. https://doi.org/10.1207/S1532690XCI2102_01.
Thomm, E., & Bromme, R. (2016). How source information shapes lay interpretations of science conflicts: Interplay between sourcing, conflict explanation, source evaluation, and claim evaluation. Reading and Writing: An Interdisciplinary Journal, 29, 1629–1652. https://doi.org/10.1007/s11145-016-9638-8.
Van Boxtel, C., & van Drie, J. (2012). “That’s in the time of the Romans!” Knowledge and strategies students use to contextualize historical images and documents. Cognition and Instruction, 30, 113–145. https://doi.org/10.1080/07370008.2012.661813.
Wiley, J., Goldman, S. R., Graesser, A. C., Sanchez, C. A., Ash, I. K., & Hemmerich, J. A. (2009). Source evaluation, comprehension, and learning in Internet science inquiry tasks. American Educational Research Journal, 46, 1060–1106. https://doi.org/10.3102/0002831209333183.
Wiley, J., & Voss, J. F. (1999). Constructing arguments from multiple sources: Tasks that promote understanding and not just memory for text. Journal of Educational Psychology, 91, 301. https://doi.org/10.1037/0022-0663.91.2.301.
Wineburg, S. S. (1991). Historical problem solving: A study of the cognitive processes used in the evaluation of documentary and pictorial evidence. Journal of Educational Psychology, 83, 73. https://doi.org/10.1037/0022-0663.83.1.73.
Wolfe, M. B., & Goldman, S. R. (2005). Relations between adolescents’ text processing and reasoning. Cognition and Instruction, 23, 467–502. https://doi.org/10.1207/s1532690xci2304_2.
Zembal-Saul, C., Munford, D., Crawford, B., Friedrichsen, P., & Land, S. (2002). Scaffolding preservice science teachers’ evidence-based arguments during an investigation of natural selection. Research in Science Education, 32, 437–463. https://doi.org/10.1023/A:1022411822951
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1: interest and attitudes
Interest Interest was assessed at the topic level. In particular, students were asked to rate their interest in each of seven topics related to gifted education (e.g., gifted and talented education, school policy, special education) using a seven point Likert scale, ranging from not at all interested to very interested. Cronbach’s alpha reliability for the seven item scale was .86. Students’ average interest ratings were 4.74 (SD = 1.07).
Attitudes Students were asked to report attitudes toward gifted education in three primary ways, capturing attitude content, stance, and certainty. Attitude content was captured by asking students to rate their agreement with 10 items reflecting statements in support of or in opposition to gifted and talented education. Statements read: Gifted education is an important priority for all schools and All students are gifted in different ways. It is wrong to prioritize some students over others. Scale reliability for the five items supporting the inclusion of gifted education in schools was .72, while reliability for the five items opposing gifted education was .70.
Students were additionally asked to adopt an attitude stance regarding the inclusion of gifted education in schools. In particular, students were asked to respond to the multiple choice item: do you think elementary schools should have gifted and talented programs and asked to select (a) yes, (b) no, or (c) I don’t know. While 56.94% (n = 41) of students supported the inclusion of gifted education in schools, 13.89% of students opposed this measure, and 29.17% indicated that they did not know whether or not they supported gifted education. Finally, students were asked to report attitude certainty. Specifically, after responding to the multiple choice item capturing attitude stance, students were asked to rate three Likert items assessing the strength of their chosen position (i.e., how sure are you? how certain are you? how strongly do you feel?). Reliability for the three item certainty scale was .92.
Across these various individual difference measures we were able to verify both that students had at least a moderate degree of interest in the topic of the task and held varied attitudes supporting and opposing gifted education. Additionally, neither interest (p = .79) nor any of the attitude-related measures (p > .07) were found to significantly differ across task conditions.
Appendix 2: sample coding of students’ notes
Appendix 3: sample response
-
1.
I believe that my school should not have a gifted and talented program. (position)
-
2.
Within the six resources I have just read, I came to the conclusion that three of the sources authors were against gifted education programs, and three were for them. (inter-text)
-
3.
Susan Moore, a teacher with a mixed abilities classroom was the first I saw to be for these programs. (organization)
-
4.
However, her arguments were completely biased and opinionated. (transform)
-
5.
She stated that teaching her mixed ability class was very “challenging and disappointing.” (text-direct)
-
6.
She further explains that this is because she cannot keep the attention of all of her students throughout the day. (intra-text)
-
7.
Personally, I think she just does not sound like the best teacher. (transform)
-
8.
If she were a good teacher, she would be able to differentiate instruction effectively so she can keep all of her students engaged and attentive. (transform)
-
9.
Frederick M. Hess was the next author to be for gifted and talented programs. (organization)
-
10.
He believes that “schools exist to provide resources and instruction that can nurture the gifts that students possess.” (text-direct)
···
-
13.
Hess also states that is its “unreasonable to expect most teachers to differentiate instruction well.” (intra-text)
-
14
Teachers are taught specifically how to differentiate instruction, so why would they not be held to that standard to do so. (transform)
-
15
Going back to Moore’s above position, good teachers will be able to differentiate instruction. (inter-text)
···
-
20
Halley J. Potter was an author who is against gifted and talented programs, and she made points that I have never thought of. (transform)
-
21
She explains that programs like these increase socioeconomic and racial segregation. (text-direct)
-
22
She gives evidence from 2011 by saying “roughly 70 percent of all New York City public school students were black or Latino, but more than 70 perfect of kindergartners in gifted and talented programs were white or Asian.” (text-direct)
-
23
If this holds true, it really is segregating schools. (elaboration)
-
24
I am not saying the schools are segregating students on purpose, however, it is still a trend from gifted and talented programs. (elaboration)
···
Rights and permissions
About this article
Cite this article
List, A., Du, H. Reasoning beyond history: examining students’ strategy use when completing a multiple text task addressing a controversial topic in education. Read Writ 34, 1003–1048 (2021). https://doi.org/10.1007/s11145-020-10095-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11145-020-10095-5