Integration refers to the formation of meaningful connections between and among disparate sources or pieces of information. Integration is implicated when students try to make sense of conflicting explanations for historical events (Britt and Aglinskas 2002), to understand the causes of scientific phenomena (Wiley et al. 2009), or even to make informed medical decisions about courses of treatment (Stadtler et al. 2014). Indeed, integration has been identified as a critical competency, necessary for learners to manage and benefit from the super-abundance of information characterizing life in the twenty-first century (List and Alexander 2017a, 2017b; Goldman and Scardamalia 2013; Lankshear 1999). Consistent with its importance, much work has examined integration, particularly when learners are presented with disparate sources of information or with multiple texts (Anmarkrud et al. 2014; Wiley and Voss 1999; Wolfe and Goldman 2005). Missing from these analyses has been an understanding of how students form connections across texts or the cognitive processes underlying integration. The goal of this study is to draw on verbal report data, gathered during students’ completion of a multiple text task, to validate a previously proposed four-level framework of integration, suggesting that students may generate connections among texts at four different levels of quality, specificity, and elaboration. As a note, the term integration (or integrative reasoning) is purposefully used throughout this paper to distinguish the processes involved in cross-textual connection formation during reading, from those involved in students’ comprehension of individual texts within a document set (Barzilai et al. 2018).

Theories of integration

A number of theories, models, and frameworks have identified the essential role of integration in students’ learning from multiple texts (e.g., List and Alexander 2018, 2019; Britt et al. 1999; Goldman 2004; Goldman et al. 2012; Perfetti et al. 1999; Rouet and Britt 2011). For example, the process-focused Multiple Document Task-Based Relevance Assessment and Content Extraction (MD-TRACE) model positions students’ formation of an integrated cognitive representation of multiple texts as the culmination of students’ development of a task model (i.e., a cognitive representation of task demands), determination of an information need (i.e., a reason to access multiple texts), and multiple text processing, including the selection of texts and evaluation of texts’ relevance (Rouet and Britt 2011). List & Alexander, 2017; 2018 suggest that students’ formation of an integrated representation of multiple texts is conditional, in part, on a variety of individual difference factors, including students’ motivations for task completion (i.e., affective engagement) and habits and skills with regard to multiple text evaluation and integration (i.e., behavioral dispositions). Likewise, considering individual difference factors, Goldman (2004) considers how students may draw on their prior knowledge in assimilating or integrating information from multiple texts, which, in turn, may be fragmented or more coherent (i.e., well-connected) in nature.

Nevertheless, in this study, we draw primarily on the Documents Model Framework in understanding how students form connections across multiple texts (Britt et al. 1999; Perfetti et al. 1999). We do so for two primary reasons. First, because the Documents Model Framework, in our estimation, provides the most detailed and systematic analysis of the types of models or cognitive representations that students may construct based on information presented across multiple texts. Second, because we view this model as foundational, with later work complementing and supplementing this to consider how integration comports with students’ broader process of multiple text use (e.g., information access, response composition, Rouet and Britt 2011), is associated with various individual difference factors (e.g., interest, Authors 2018, 2019), and manifests variably across domains (e.g., history, science, Goldman et al. 2012). Still, we recognize the important role of all of these conceptualizations of integration in our understanding of this construct (see List & Alexander (2018) and Barzilai et al. 2018 for overviews).

In the Documents Model Framework students are considered to integrate multiple texts by forming two representations or mental models of texts: the integrated mental model and the inter-text model (Britt et al. 1999; Perfetti et al. 1999; Rouet and Britt 2011). The integrated mental model is a unified understanding of the common situation or issue discussed across texts. The inter-text model is a structural model that represents two types of relations. These are the relations between content and information about its source of origin (e.g., document information like author or publisher) and the relations among sources, as agreeing, disagreeing, or otherwise complementing one another. As further described by Britt et al. (1999), once students form integrated mental models and inter-text models of multiple texts, these may be linked in various ways (i.e., by students constructing mush models, separate representations models, or documents models of multiple texts).

The mush model is characterized by students’ effective construction of an integrated mental model, at the same time that inter-text model development is limited (Britt et al. 1999). This means that students constructing mush models are able to successfully integrate information presented across texts; however, they are limited in tracing particular pieces of information back to their sources of origin. As a result, when there are discrepancies or conflicts arising in the information to be integrated, students are unable to compare its sources of origin as a means of conflict resolution. Conversely, students constructing a separate representations model are limited in their integrated mental model construction but they may be successful in formulating inter-text models of multiple texts. This means that while students may be able to link particular pieces of information back to their sources of origin, they fail to form connections across texts or to develop a coherent representation of the common issue or topic described.

When students are able to successfully connect their integrated mental models and inter-text models, they form a documents model of multiple texts. The documents model is characterized by its connecting or integrating of content from across texts and by its tagging or linking of specific content to source of origin. This means that in the documents model, common information, presented across texts, should, ideally, be emphasized, while discrepant or conflicting information is evaluated and reconciled. Indeed, the documents model should integrate not only the content presented across texts (in the integrated mental model) but also the sources themselves, through a mapping of the relations among them.

While the Documents Model Framework is effective in describing the types of integration that may occur as an outcome of multiple text use or may manifest in the written responses that students compose, it does less to identify the processes whereby such integration may occur. It is this process of integration that we principally seek to examine in this study. In particular, the process of integration (i.e., integrative reasoning) may be understood as students’ formation of cross-textual connections during reading and the specific cross-textual connections that result; with such integrative processing being distinct from the mental models or cognitive representation of multiple texts that students ultimately form, which may be integrated (i.e., reflecting a documents model) or not List & Alexander (2019).

Process of integration

Kurby et al. (2005) suggest that students’ multiple text integration may be the result of both top-down and bottom-up processes. Bottom-up or resonance-based integration processes are those that occur automatically as a result of the semantic or conceptual overlap between texts. Kurby et al. (2005) refer to these as data-driven or “dumb” mechanisms of integration. Top-down or evaluative processes of integration represent deliberate efforts on the part of learners to form connections across texts. Kurby et al. (2005) used reading time and performance data to examine the role of bottom-up and top-down processes in integration. Based on two experiments they concluded that students engage in integration, even without explicit instructions to do so, by using information in one text to understand ambiguous information in a second. At the same time, they determined that top-down processes typically take precedence in dictating integration, except for in cases where there is a great deal of semantic and conceptual overlap between texts. Two questions stemming from Kurby et al.’s (2005) work are how students coordinate their use of top-down vis-à-vis bottom-up processes to support integration and how students’ deliberate, top-down process of integration manifests.

Drawing on Kurby et al.’s (2005) earlier work, as well as Britt et al.’s (1999) description of documents model formation, List & Alexander (2019) recently proposed that students may demonstrate four levels of integration, or integrative reasoning, when forming connections across texts. Here, we use the term integration and integrative reasoning to capture both students’ process of connection formation during reading and the (cognitive) connections that are formed. As such, we conceptualized integration during text processing in much the same way that prior work has viewed inferencing (McNamara et al. 2004), as both the cognitive outcomes generated (i.e., an inference or a single cross-textual connection) and the underlying processes (i.e., inferencing or integration) involved. Each of the four levels of integration, specified by List & Alexander (2019), may be considered to reflect a distinct cognitive representation of a cross-textual connection, differing in elaboration and quality.

Level 1 or relational identification involves students noticing or attending to a potential overlap or connection across texts. This may be result of the semantic overlap between texts (i.e., associated with bottom-up integration processes) or of students’ more deliberate search for cross-textual connections (e.g., associated with top-down integration processes). Level 2 involves students separately understanding potentially related information presented within each individual text, referred to as the separate representation of information. Third, students link texts to one another via a single, compound statement using a connective term (e.g., and, however). Such linking requires the formation of a novel, super-ordinal or overarching connection, not explicitly introduced in texts and may best correspond to intertext predicates, as defined by Perfetti et al. (1999). This level of integration is referred to as students drawing a simultaneous relation across texts and involves the specification of a relationship across these. Finally, in level 4 or relational elaboration (previously referred to as relationship designation), students elaborate on or further explain the relationship identified in the previous level.

These final two levels include students’ designation of a relationship across texts and its explanation. While a variety of possible cross-textual relationships may be formed, at minimum, these include potential corroborative, comparative, conflicting, and causal connections developed across texts (Du & List, (2020) and List, Du, Wang, & Lee (2019); Anmarkrud et al. 2014; Primor and Katzir 2018; Wolfe and Goldman 2005). Regardless of the types of connections formed, these may be explained or elaborated to varying extents, corresponding to level 4: relational elaboration. Across these four levels of integration, while bottom-up processes may determine students’ initial attendance to potential relations across texts, top-down processes, or effortful attempts at integration, are likely needed to form higher-level relations across texts. As such, these four levels are conceptualized as not only taxonomic but also progressive in sophistication, at least to some extent. We further consider it likely that students need to form lower levels of integrative connections before higher level relations are able to be drawn.

At present, these four levels of integrative reasoning have only been theorized to occur during students’ multiple text use, with some emergent evidence for these different levels of integration quality reflected in students’ writing (List, Du, Wang, & Lee, 2019). Nevertheless, the extent to which such integrative reasoning is present in students’ processing of texts remains an open question. Indeed, Barzilai et al. (2018) in a recent review of intervention studies targeting multiple text integration found 96.7% of these to employ writing tasks to assess integration, with only 8.2% of these employing think-alouds as process measures of integration. This is echoed in Primor and Katzir’s review of integration (2018) which found only two studies to collect process data on integration, leaving questions regarding how students form cross-textual connections during multiple text processing.

Nevertheless, the limited think-aloud studies that have been conducted have found cross-textual connection formation to be well-represented in students’ multiple text processing (Cerdán and Vidal-Abarca 2008). For instance, Anmarkrud et al. (2014) found 52.7% of the strategies that students engaged during multiple text processing to reflect the elaboration of information within one text by connecting this with information presented in other texts. While highlighting the prominent role that integration or cross-textual elaboration plays in students’ strategy use when learning from multiple texts, Anmarkrud et al. (2014) did not further interrogate the quality of students’ elaborations or cross-textual connections formed, as we aim to do in the present study.

Wolfe and Goldman (2005), in a study of sixth grade students thinking-aloud while reading two isomorphic texts presenting conflicting causes for the fall of the Roman Empire, attempted to classify the quality of students’ integration during processing. They did so by separating students’ cross-textual comments as reflecting either surface text connections or self-explanations. Surface text connections were based on superficial features of texts and reflected students forming: “a compound or complex sentence out of two separate text sentences” (p. 481). As a contrast, self-explanations were statements aimed at improving understanding through the connecting of information across texts; these commonly reflected students’ formation of causal and comparative cross-textual connections. Despite their difference in sophistication, both surface text connections and self-explanations were found by Wolfe and Goldman (2005) to be associated with students’ task performance (i.e., quality of historical reasoning).

In this study, we build on Wolfe and Goldman’s (2005) work to further examine gradations or degrees of sophistication in students’ multiple text integration during processing (i.e., integrative reasoning). In particular, we examine the nature of students’ multiple text integration occurring between students’ formation of surface text connections (e.g., potentially corresponding to level 1: relational identification) and self-explanations (i.e., potentially reflective of level 4: relational elaboration). We further examine verbal report data to validate the four levels of integrative reasoning described by List & Alexander (2019).

Present study

In this study, we adopt a within-subjects design to examine the extent to which students demonstrate four levels of integrative reasoning, across two types of integration tasks, one more elemental and the other more complex. In the first integration task, we examine students’ connection formation across two parallel, yet conflicting, texts presenting contrasting explanations for the construction of Stonehenge (i.e., elemental task). However, in recognition of the fact that multiple texts are often times not written to be isomorphic with one another and able to be contrasted directly, we further examine integration when students are asked to complete a more challenging and ambiguous multiple text task—to form connections across eight texts offering a number of different perspectives for and against the legalization of sex work (i.e., complex task).

Further, we examine how students integrate these two sets of texts at three different levels of specificity: forming (a) evidentiary, (b) thematic, and (c) contextual connections across texts. Drawing on work by List, Du, Wang, & Lee (2019), we examine the extent to which students form connections between specific evidence presented across texts (i.e., evidentiary integration), the main ideas or claims introduced (i.e., thematic integration), and cross-textual meta-document features, like authors’ trustworthiness (i.e., contextual integration). We examined students’ cross-textual connection formation across these three different levels of specificity to align with work that has analytically conceptualized texts as having a source or author and content, including main ideas and supporting evidence or information (Britt et al., 2012). Indeed, Perfetti et al. (1999) in the DMF explain that integration involves students’ connection formation across texts or document nodes, with each document node constituting a representation of a text, including its (a) source and source-related information, (b) the source’s rhetorical goals (i.e., author intent), and (c) content. In line with such analytic conceptions of texts, we examine students’ cross-textual connection formation across all of these various text-based elements. We do this to determine how students link texts or which text-based elements students attend to in forming source-source and content-content links.

Finally, we included two outcome measures, one corresponding to each task, to examine whether the nature of students’ cross-textual connection formation during text processing resulted in improved integration task performance. For the elemental integration task, we used an evidence sort task that asked students to correctly identify information, presented across texts, as uniquely introduced within one text or another, or as commonly introduced across texts. For the complex integration task, students were asked to create a diagram or a visual representation of the multiple texts provided and the connections among them.

These two performance tasks were used in this study for three primary reasons. First, these were outcome measures that allowed us to examine any association between students’ integrative reasoning during reading and holistic integration performance (i.e., requiring the linking of all of the texts introduced within a document set). That is, the evidence sort task required students to categorize the evidence present across both of the argument texts provided, while the diagram construction task explicitly directed students to connect all eight of the texts they read for the complex integration task. As such, in requiring students to connect all of the texts provided, in some way, both of these tasks stand in contrast to prior assessments which have either allowed students to self-select which information from texts to integrate (e.g., when writing) or have focused on more local text connections (e.g., evaluate integrated statements, reflecting connections only among two or three texts). Second, both of these tasks allowed students to link texts according to content (i.e., form content-content links) and source. In the case of the evidence sort task, students both had to link common evidence together (i.e., form content-content links) and to separate particular evidence as coming from the distinct texts provided (i.e., form source-content links). In the case of the diagram construction task, students were able to form content-content, source-source, and source-content links in representing connections among texts. As a final point, these two tasks were selected because these were atypical, both in terms of the types of assignments that students may be asked to complete for class and in terms of common methods used to assess integration (i.e., writing, Barzilai et al. 2018). We thought using non-writing based assessments to tap integration would allow students to focus on cross-textual connection formation, without the associated demands of written response composition List (2019).

As a note, the evidence sort task was used in association with the elemental integration task, while the diagram task was used in association with the complex integration tasks because these were thought to correspond to the structure of the text-sets used for each task. That is, the evidence sort task was intended to reflect the isomorphic structure of the two argument texts, while the diagram construction task was intended to reflect the intertwined nature of the text set used for the complex integration task. Both the evidence sort task and the diagram construction task have been used to capture integration in prior work (e.g., Author).

We have the following research questions:

  1. 1.

    What kinds of cross-textual connections do students form when asked to relate a set of isomorphic texts presenting conflicting information?

  2. 2.

    What kinds of cross-textual connections do students form when asked to relate a set of texts introducing a variety of perspectives in support of and in opposition to the legalization of sex work?

  3. 3.

    For each task, what is the association among students’ reports of cross-textual connection formation following reading and integration task performance?

  4. 4.

    To what extent is integration performance associated across tasks?

Methods

Participants

Participants were (N = 33) undergraduate students at a large university in the Northeastern USA (age: M = 19.39, SD = 1.20). The sample was 75.76% female (n = 25; male: 24.24%, n = 8). The majority of the sample was White (75.76%, n = 25), with 9.09% of students reporting Asian (n = 3) and mixed race (n = 3) ethnicity. Participants were recruited from three human development courses and offered extra credit for participation. All participants had to be at least 18 years old to participate.

Procedures

First, students were asked to complete topic-specific assessments of prior knowledge and respond to a question asking them about their stance on whether or not sex work should be legalized. These individual difference measures were only examined descriptively to better characterize participants in our sample. For instance, the prior knowledge measure served only to confirm the novice or low prior knowledge nature of our sample. Then, students were asked to complete two multiple text tasks, one more elemental and the other more complex, in counter-balanced order. All data were collected one-on-one, in a quiet study room. Data collection sessions lasted approximately one hour.

Individual difference measures

Prior knowledge

Topic-specific prior knowledge was assessed by asking students to respond to open-ended questions associated with each topic. For the elemental integration task, the prior knowledge question read: One of the topics we will ask you about is Stonehenge and why it was constructed. Please tell us everything you know about Stonehenge. For the complex integration task, the prior knowledge question read: Please list any words, terms, or concepts that you associate with legalizing prostitution/sex work. Try to list as many terms and concepts as you can. Both questions were scored for the number of relevant ideas these included. Average prior knowledge scores for the elemental integration task were M = 0.81 (SD = 1.35), while average prior knowledge scores for the complex integration task were M = 1.90 (SD = 1.21). Prior knowledge scores were only considered descriptively to ascertain the novice nature of our sample. Inter-rater reliability for prior knowledge, based on two raters scoring all student responses, was 83.87% exact agreement for the elemental task and 84.85% exact agreement for the complex task. Disagreements were resolved through discussion.

Attitudes

Participants were also asked to report their attitude stance on the topic of legalizing sex work. In particular, participants were asked to respond to the prompt: In this study, we will ask you to think about the topic of legalizing sex work/prostitution. Do you think that sex work/prostitution should be legalized? Participants were asked to select from four options: yes (72.73%, n = 24), no (9.09%, n = 3), I’m not sure (12.12%, n = 4), and I don’t know enough to decide (6.06%, n = 2). Attitude data were examined descriptively to ascertain the controversial nature of the sex work topic, used for the complex integration task.

Multiple text integration tasks

Elemental integration task

The elemental integration task asked students to read a set of three texts presenting conflicting accounts for why Stonehenge was constructed.

Stonehenge texts

Students were presented with a set of three texts on the topic of Stonehenge. The first text (i.e., introductory text) provided general information about the history and construction of Stonehenge. The two subsequent texts (i.e., argument texts) were constructed to provide two conflicting explanations for why Stonehenge was constructed. Specifically, each argument text introduced a central claim (i.e., that Stonehenge was constructed either as a burial site or as a center for healing), followed by five pieces of supporting evidence (e.g., human remains found in the area). Across the two argument texts, two pieces of evidence were common or shared, although variably interpreted within each text, while three pieces of evidence were unique, appearing only in one of the texts provided.

All texts were presented to students on separate sheets of paper with, title, author, and publisher information listed at the top of each page. For the elemental task, these three texts were introduced to students as excerpts from the same textbook, with each of the argumentative texts formatted to begin with a distinct secondary endogenous source (Strømsø et al. 2003). That is, the first text introduced the controversy with the statement: Today there are two major theory about why Stonehenge was built. Then, each of the two conflicting argument texts began with a distinct, secondary endogenous citation (e.g., Mike Parker Pearson, archeologist from the University of Sheffield in England argues that Stonehenge was used as a burial site). Texts were matched for length and readability (see Table 1). All students first received the introductory text about Stonehenge but were presented with the subsequent two argument texts in counter-balanced order. The elemental integration task had at least six cross-textual connections embedded within it for students to identify (i.e., (1) identifying that all three texts were excerpts from the same textbook, (2) but that the two secondary endogenous sources were experts, whose views conflicted with one another; (3) identifying that the controversy around why Stonehenge was built, in Text 1, was explained in Texts 2 and 3, (4) recognizing that Text 2 and Text 3 conflicted with one another in their explanations for Stonehenge’s construction, and (5 and 6) identifying each piece of common evidence as shared across the two argument texts). Nevertheless, a number of additional connections, at various levels of specificity, could be identified (e.g., both Text 1 and Text 3 discussed the trilithons that Stonehenge is composed of).

Table 1 Word count and Flesch-Kincaid grade level across texts
Full size table

Instructions

When students were provided with the set of three texts, they were further instructed to read these in order to form an argument about why Stonehenge was constructed. Moreover, students were told that, as they were reading, they may notice some connections or relations across texts. Students were asked to highlight each connection or relation that they noticed as well as to verbally explain the connection into the tape recorder provided. In particular, students received the following instructions verbally and written on a directions sheet prior to reading:

Please read the texts in this packet. They provide conflicting explanations for why Stonehenge was constructed. Read these texts to decide why you think Stonehenge was constructed.

As you read, you may notice some connections or relationships between the texts. Every time you see a connection between the texts, please highlight it and explain the connection out loud.

In these instructions, students were specifically asked to “explain the connections” they provided because we wanted to elicit as much explanation and elaboration as possible for the cross-textual connections that students formed. In fact, we expected the process of verbalizing cross-textual connections, alone, to foster such explanation and elaboration. Students were provided with highlighters, markers, and different color pens to aid in the identification of relations across texts. Students’ identified cross-textual relations and associated explanations are the primary focus of the present analyses.

Evidence sort task

After students completed reading and identifying relations across texts, they were asked to complete an evidence sort task. This task presented students with index cards listing the eight pieces of evidence that appeared across the two argument texts (i.e., two common, six unique). Students were asked to sort each piece of evidence according to whether it was introduced in only one of the argument texts (i.e., Stonehenge being constructed as a burial site or as a center for healing) or discussed in both of the argument texts provided. The evidence sort task was used as a criterion variable in this study to examine the extent to which students’ connection formation across texts was associated with task performance.

Initially, the evidence sort measure had an unacceptably low degree of reliability. By examining item-scale statistics, two items were excluded, resulting in an internal consistency of Cronbach’s α = 0.52. Among the two items removed, once represented a piece of common evidence while the other pertained to the text describing Stonehenge as a center for healing. While this represented a poor degree of reliability, this measure may not have represented a unidimensional construct but rather students’ separate representations of two texts and the relations between them.

While students had the texts available to them when reading and identifying relations across texts during the elemental task, texts were taken away when students completed the evidence sort task.

Complex integration task

The complex integration task asked students to read and identify relations across eight texts presenting a variety of complementary and conflicting views on the topic of legalizing sex work.

Sex work texts

Students were presented with eight texts introducing a variety of perspectives arguing for and against the legalization of sex work. In particular, texts varied in presenting public health, economic, legal, feminist, and moral perspectives in support of and in opposition to this issue. Texts were drawn from the Room for Debate segment of the New York Times: https://www.nytimes.com/roomfordebate/2012/04/19/is-legalized-prostitution-safer. Texts were modified for inclusion in this study. Information about all eight texts is provided in Table 1.

Each text was presented to students on a separate, different colored, piece of paper, in a random order. Each text included a title at the top, with author information and credentials following. All texts were matched for length and readability. Figure 2 included a sample annotated text, with connections indicated.

Instructions

As with the elemental integration task, the complex integration task also asked students to read the texts provided to form an argument for or against the legalization of sex work. Again, students were asked to identify (i.e., to highlight) as many relations across texts as they could and to explain these. In particular, students received the following instructions both verbally and written on a directions sheet, prior to being presented with texts for the complex integration task:

Please read the texts in this packet. They provide different points of view on whether or not sex work/prostitution should be legalized. Read these texts to decide whether or not you think sex work/prostitution should be legalized.

As you read, you may notice some connections or relationships between the texts. Every time you see a connection between the texts, please highlight it and explain the connection out loud.

Beyond these initial task instructions, presented prior to reading, students were not further prompted to identify connections as they read. However, once students had indicated that they were done identifying connections across texts, they were asked: Are there any other connections or relationships between texts that you want to identify. Only once students had indicated that they had no further connections to identify, did they proceed to the diagram construction task. An example of students’ annotations, identifying connections across texts, is included in Fig. 1.

Diagram construction task

The diagram construction task asked students to create a diagram or a visual representation of the eight texts provided and the relations among them. Students were provided with a blank sheet of cardstock as well as with a variety of colored pens and markers to aid in diagram construction. See Fig. 2 for a partial representation of the variety of connections able to be drawn across texts. Although a definitive number of cross-textual connections available for students to form for the complex integration task could not be identified, researchers identified over 50 distinct cross-textual connections, while a prior study (Author, 2019) using the same materials, but only asking students to form a diagram or visual representation of multiple texts, found students to form up to 18 cross-textual connections in the diagrams they constructed.

Fig. 1
figure 1

Sample text with cross-textual connections indicated

Full size image
Fig. 2
figure 2

Partial representation of the connections among texts opposing the legalization of sex work

Full size image

Diagrams were coded for the type of multiple text models represented. The multiple text models that students depicted were differentiated according to whether or not they made explicit reference to the texts provided (e.g., by title or by author) and whether or not they formed any connections across texts. For instance, representations that connected information presented across texts (e.g., linking texts in terms of their consideration of worker health and safety) were categorized as either mush models or documents models of multiple texts. The latter code was applied to representations that also explicitly labeled information according to its source of origin. The four different types of multiple text models coded for are presented in Fig. 3. Cohen’s kappa inter-rater agreement for scoring all students’ diagrammatic representations of multiple texts was κ = .92 (exact agreement: 93.94%, n = 31), indicating near perfect agreement. Students had the eight texts available to them throughout their completion of the complex integration task, during both their formation of cross-textual relations across texts and diagram construction task completion.

Fig. 3
figure 3

Models reflected in students’ graphic representations of multiple texts

Full size image

Coding integration

Instances of integration were first identified by reconciling students’ verbal reports with the highlights and annotations that they made on the texts provided. Instances of integration were primarily segmented based on transcriptions of students’ cross-textual connection formation. That is, in their audio recordings students typically paused between identifying cross-textual connections. These pauses were indicated in the transcripts and were used for segmentation. In instances when segmentation was ambiguous (i.e., when students were identifying multiple connections in the same statement), instances of integration were segmented by corroborating these against students’ annotated texts (e.g., whether students highlighted the same information in two different colors).

Each instance of integration, after being segmented, was coded in four ways. First, each instance was coded according to the texts it implicated. Second, each instance of integration was coded for the level of specificity it reflected. This included connections drawn at the evidentiary (i.e., linking specific information across texts), thematic (i.e., connecting main ideas across texts), and contextual (i.e., comparing author information and other meta-textual features) levels.

Third, instances of integration were coded according to the type of cross-textual connection they specified. Connections identified most commonly reflected instances of agreement or disagreement across texts. However, other types of connections, including those that were explanatory (i.e., using information from one text to understand information in another text) or inferential (i.e., using information in one text to elaborate or predict information in another) in nature, were also identified.

Finally, instances of integration were coded for the level of integrative reasoning they reflected. Following from List & Alexander (2019), four levels of integrative reasoning were identified (i.e., level 1, relational identification; level 2, separate representation; level 3, simultaneous relation; level 4, relational elaboration). Examples of responses reflecting each level of integration are included in Table 2. Consistent with prior work, we conceptualized these levels of integration in a progressive, rather than categorical manner as, from level 1 to level 4, students’ quality of connections formed across texts increased in specificity (i.e., level 2 vis-à-vis level 1), abstraction (i.e., level 3 vis-à-vis level 2), and elaboration (i.e., level 4 vis-à-vis level 3). We expected each increased level of integration to reflect learners’ richer, more complete, and more robust cognitive representation of a connection among two or more texts.

Table 2 Sample connections reflecting each level of integration
Full size table

Each response was coded according to these four dimensions. For instance, a response such as: “Both texts mentions these trilithons as doorways or gateways,” was coded as (a) linking two texts, (b) forming an evidentiary connection, regarding the mention of a specific structure, (c) as capturing cross-textual agreement, and (d) as reflecting the first level of integration or only the semantic overlap of information presented across texts (i.e., relational identification). Another response was, “The fourth and fifth articles give different statistics and information based on how these workers feel, the fourth one mentioning that they feel uncomfortable and that they’re stuck and that they’re unsafe, while the fifth article refers to them feeling that they’re only bound by their contract and that they do feel safe.” This response was also coded as (a) linking two texts and as (b) forming an evidentiary connection regarding the specific information presented across them. At the same time, this response was also coded as identifying an instance of (c) disagreement and as (d) corresponding to level 4 of integration (i.e., relational elaboration). In this response, not only was a novel, higher-order connection identified across texts (i.e., “the fourth and fifth article give different statistics”) but this connection was further elaborated based on the specific statistics introduced within each text.

Two raters scored each instance of students’ connection formation across the two tasks. Inter-rater agreement for the number of texts included in each connection that students formed was 95.16% (n = 118) for the elemental task and 96.13% (n = 273) for the complex task. Exact agreement was 91.94% (n = 114) for the specificity of connections identified, in the case of the elemental task, and 94.01% (n = 267) for the complex task. In terms of the types of connections formed across texts, exact agreement was 93.55% (n = 116) for the elemental task and 94.72% (n = 269) for the complex task. Finally, exact agreement for the level of elaboration reflected in students’ connection formation was 79.03% (n = 98) for the elemental task and 77.82% (n = 221). All disagreements were resolved through discussion.

Results

Research question 1: relation formation for the elemental integration task

The first research question examined the nature of students’ cross-textual relation formation when completing the elemental integration task. Descriptives are included in Table 3.

Table 3 Number of relations formed across multiple text tasks
Full size table

Number of relations

In response to the elemental integration task, participants drew an average of 3.88 connections across texts (SD = 4.53), with zero to 24 connections identified.

Number of texts connected

Based on a total of 124 cross-textual connections identified, 88.71% (n = 110) linked two texts together, while 11.29% (n = 14) connected all three texts to one another. Relations connecting all three texts included: “All three texts come from the same book by the same author.”

Specificity of integration

More evidentiary (M = 2.25, SD = 3.23) than thematic (M = 1.41, SD = 1.43) connections were identified across texts. This is understandable, given that texts were designed to included five pieces of evidence within each. Contextual cross-textual connections were identified least commonly of all (M = 0.22, SD = 0.66). Instances of evidentiary integration involved participants corroborating specific information across texts, as demonstrated in responses such as: “I’m highlighting ‘for at least 500 years’ and on the first one I'm highlighting ‘4000 years and 2000 years’ just because they’re both talking about something that was built a very long time ago.” Thematic integration involved the juxtaposition of main ideas or key themes across texts. This was reflected in responses such as “This one says it was used as a burial site, and the last one did too,” referring to one of the two conflicting explanations provided for Stonehenge’s construction. Finally, instances of contextual integration across texts occurred when participants compared author characteristics. For example: “For the second and third article, both of the researchers are from England,” constituted an example of contextual integration.

Type of integration

In terms of the types of connections identified, most commonly these noted agreement or consistency across texts (M = 3.06, SD = 4.23). Identifications of disagreement (M = 0.19, SD = 0.47) or some other relation (M = 0.63, SD = 1.04) across texts were comparatively rarer. An example of agreement across texts included a student identifying that all three texts referred to similar structures: “And I’m highlighting ‘great wooden circle’ because in the first it said ‘arranged in a semi-circle.’ Yes, stone circle, semi-circle.” Instances of conflict identification included the following: “but it’s interesting how they had different perspectives, the two archeologists and professors.” Finally, additional connections identified were explanatory, organizational, complementary, and corroborative in nature. For instance, the statement: “But the first article relates to both of those, because it describes the two major theories, being that it's a burial site and an ancient center for healing. And then it goes into them in the other ones,” was coded as identifying an organizational relation across texts. At the same time, a statement such as: “On the second one, they say that they have found...Human remains on the sites. And that many of the remains had deformities, which was probably why they were dying and buried there,” was coded as identifying an explanatory connection across texts, as it was using information in one text to aid in making sense of another.

Level of integration

Finally, the level of cross-textual integration demonstrated was coded for. Most commonly students engaged in relational identification (M = 2.03, SD = 3.07). This was reflected in responses such as, “Two of them say that...stone chips have been found in the area surrounding Stonehenge.” Such relations were based only on the semantic overlap between information presented across different texts, with no further attention to the purpose or significance of this inter-relation. Separate representation, or the explicitly identification of related information within each text, was reported next most commonly (M = 1.03, SD = 1.43). This involved students making statements such as: “One article talks more about how it was for healing while the other one talks about the Stonehenge being a burial site.” In this case, while specific information included in each text is explicitly identified, still no higher-order connection is identified across these. That is, students are not providing a conceptual reason for the linking of these texts.

Higher levels of integration were comparatively rarer. Specifically, simultaneous relation (M = 0.41, SD = 0.80) and relational elaboration (M = 0.41, SD = 0.87) each reflected only 10.48% (n = 13) of relations identified. Instances of connection formation classified as reflecting a simultaneous relation occurred in students’ responses such as, “Both of the second articles, the one that talks about healing, and the one that talks about a burial site evidence...they got evidence from the same place, from excavations.” This response was coded as reflecting a simultaneous relation because it linked information from across texts together into a single, conceptual statement (i.e., both of the second articles) and because it explained the function of linking information from across texts (i.e., “they got evidence from the same place”). Such a statement stands in contrast to students who may have identified that both authors examined excavations. While the use of excavations was explicitly stated in the texts provided, the fact that authors used these as a common data source was a novel, higher-order association generated by the learner. The use of excavations as common evidence provides a reason to link this information across texts, rather than simply attending to the word “excavations” being commonly used across texts. Finally, responses such as: “The second one talks about things being buried… alongside the human remains. The third one also brings it up, but more specifically, it talks about domestic objects including wood or jugs and woven baskets. They name more specific stuff,” were classified as reflecting relational elaboration. In responses such as this, not only was a novel, higher-order connection identified (i.e., “brings it up, but more specifically) but this connection was also elaborated (i.e., “they name more specific stuff”).

Research question 2: relation formation for the complex integration task

The second research question examined students’ connection formation when presented with the complex integration task. Descriptives are included in Table 3.

Number of relations

On average, students identified 8.61 (SD = 12.23) cross-textual connections, with zero to 57 connections formed.

Number of texts connected

Among the 284 connections formed, 56.69% (n = 161) linked two texts to one another, while 43.31% (n = 123) of relations connected more than two texts. A connection linking more than two texts together included: “I’m going to highlight ‘46 [percent] reduction’ and like just all the statistics from the first one, because in the first article they said like ‘It’s hard to explain the demographics,’ but in the last two [texts], they gave me a lot of statistics.”

Specificity of integration

We then looked at the specificity at which texts were connected to one another. Connections formed were most commonly evidentiary (M = 4.91, SD = 6.33), followed in frequency by thematic (M = 3.58, SD = 7.18) and contextual (M = 0.12, SD = 0.33) connections formed. In fact, while 75.76% (n = 25) of students formed at least one evidentiary connection across texts, only 12.12% (n = 4) identified a contextual cross-textual connection. The formation of evidentiary connections was reflected in responses such as: “They both say that women who begin enter at [a] very young [age],” comparing specific information across texts. Thematic connections across texts were reflected in responses such as:

“Okay, so in the ‘Legalizing Prostitution as a Feminist Issue,’ the woman goes on to say that prostitution being legalized is actually an act of feminism and it’s letting women control their bodies and it’s completely moral. And then, in ‘Prostitution is Immoral’, he’s, I assume it’s by a man, he’s literally saying that it’s actually immoral as it’s the opposite of feminist movements.”

Responses such as these juxtaposed the main theses or arguments introduced across texts. Finally, contextual connections further compared author characteristics or perspectives across texts. For instance, a response such as, “And that also relates… since that was a female’s perspective, ‘Trust me,’ [be]cause she was giving her own perspective. That goes along with Female Oppression in the next one, so that relates to that,” reflected a participant linking texts because of their shared female-focused perspective.

Type of integration

We further examined the types of connections that students identified across texts. Most commonly, students identified points of agreement across texts (M = 6.64, SD = 8.81), with disagreements (M = 1.18, SD = 2.51) and other types of connections (M = 0.79, SD = 1.80) less frequently identified. Indeed, 84.85% (n = 28) of students identified at least one instance of agreement across texts, as compared to only 33.33% (n = 11) of students identifying at least one disagreement. Agreements across texts were reflected in responses such as, “Okay, so this is saying as a teenager and in the first one it was saying how a lot of girls enter when they’re just 16. I’ll say you’re both talking about very young girls.” Disagreement was evidenced in responses such as:

“And then a connection between Sex Workers and Prostitution is Immoral, there actually is a connection because this author is admitting that there could be improvements, in that prostitution isn’t necessarily the safest ideal, whereas the first article, that was saying it’s completely safe. But the Prostitution is Immoral article is saying that it's not safe completely, so at least there’s a connection between these two.”

Additional types of connections identified included inferential, refutational, corroborative, and explanatory connections among texts. For instance, a response classified as inferential included: “So in Sex Work as a Civil Right it explains that the government should not throw consenting adults into jail for private sexual conduct. And it relates because it says, ‘consenting adults.’ And in the, Sex Workers Have a Right to Labor, it explains that in legal brothels, so that’s like consented, ‘employees… are bound only by their contract.’” This student is using information in one text to make an inference regarding the role of consent in legal brothels, an issue introduced in a second text. Refutations were reflected in responses such as: “So in the fourth article it says, ‘One woman told us that the brothel allowed her to sever ties with an abusive pimp.’ That connects with the first article saying that, ‘Several of us had pimps, despite working in a legal establishment.’ The issue that I have with that is how would they know that she has an abusive pimp.” In this case, the student is using information in one text to refute or question the argument introduced in another.

Level of integration

Finally, we examined the level of integration reflected in the connections that students found. Connections reflecting only relational identification, or semantic overlap, were by far the most common (M = 3.15, SD = 4.59). Such connections were evidenced in responses such as, “The first two readings both state that prostitution is not safe,” reflecting attention to only the sematic overlap of information across texts. Relations reflecting the other three levels of elaboration occurred roughly in proportion to one another: separate representation: M = 1.36, SD = 2.51; simultaneous relation: M = 2.52, SD = 5.35; relational elaboration: M = 1.58, SD = 3.60. Connections reflecting the separate representation of information included: “My next connection is between Prostitution is Female Oppression and Prostitution is Immoral. So Prostitution is Female Oppression, they have, ‘Our approach to prostitution should be premised on the idea that women and girls have the right not to be bought and sold for sexual exploitation.’ And Prostitution is Immoral says prostitution is not just selling labor, it involves the commodification and sale of women’s bodies in and of themselves…”. Such a response is an improvement on only relational identification, as it explicitly identifies the information that is linked within each text.

Responses evidencing the simultaneous relation of multiple texts included: “And then it says in the next sentence on the second article that, ‘In interviews with 66 adolescents being prostituted, Farley found, asked adolescents to explain entry into the sex trade as a way to escape from violence and abuse.’ So I’m [going to] highlight that blue for violence and abuse. ‘Including sex abuse,’ just like the first article says.” In this response not only is specific, to-be-linked information identified in each text but also a thematic relation is drawn across these (i.e., highlight that blue for violence and abuse). In this case, the student is drawing a novel, thematic connection across texts—one that is not explicitly stated. At the same time, this connection was not elaborated to a meaningful extent. Finally, responses demonstrating relational elaboration included:

“So in the third article, by Todd, he says that, ‘Legalizing prostitution allows for its regulation and creates a safe environment for sex workers and client alike.’ But that idea strongly disagrees with the first articles theme which is that even though she was a sex worker in Germany, which is legal, there was still a lot of violence involved in the industry. So basically... It disagrees with the idea, that’s what it does. Like the first article says, that no matter what, no matter how legal it is, violence will still be prevalent.”

We consider this response and others like it to be the most effective approach to multiple text integration for three primary reasons. First, this response explicitly identifies the information that is to be linked within each text. Second, this response identifies a novel, super ordinal connection across texts (“but that idea strongly disagrees). Finally, this response elaborates the nature of this connection and further explains the significance of the disagreement identified (i.e., Like the first article says, that no matter what, no matter how legal it is, violence will still be prevalent”).

Research question 3: number of connections formed and performance

Our third research question examined the association among the integrative connections that students formed following reading and their performance on each multiple text task.

Elemental integration task

For the elemental integration task, we examined the association between performance on the evidence sort task and the number of connections that students identified across texts. Spearman’s rho was used as a non-parametric alternative to a Pearson’s correlation to account for the non-normal distribution of the number of connections that students formed across texts. However, the number of connections formed were not associated with evidence sort performance to a significant extent, ρ(32) = .17, p = .35.

Complex integration task

For the complex integration task, the number of connections formed were examined across the models of multiple texts reflected in students' diagrams. A Kruskal-Wallis test was used as a non-parametric alternative to the one-way ANOVA, to account for outliers and non-normality present in the number of connections that students identified. However, the number of connections formed was not found to differ to a significant extent across multiple text models in diagrams, p = .67. Nevertheless, all means were in the directions expected (see Table 4), as were mean ranks (separate representation, with citations, 14.17; separate representation, no citations, 15.43; mush model, 16.77; documents model, 19.25).

Table 4 Number of relations formed by multiple text model for the complex task
Full size table

Research question 4: associations among integration performance across tasks

Our final research question examined students’ integration performance across the two tasks. The number of connections that students identified across tasks were found to be significantly associated with one another, ρ(32) = .54, p < .001. The correlations among the number of connections formed at each levels of integration, across tasks, are in Table 5.

Table 5 Spearman’s rho correlations among the number of connections formed at each level of integration, across tasks
Full size table

Summary of results

We examined students’ integration across two tasks, one more elemental and the other more complex (research questions 1 and 2). Across two tasks, students were found to generate more evidentiary connections than thematic or contextual connections and to identify more similarities than differences or other types of relations across texts. In terms of the levels of integrative reasoning exhibited, students formed more lower-level than higher-level connections. Additionally, while students’ integrative reasoning during text processing was not found to be significantly associated with task performance (research question 3), integrative reasoning was associated across the elemental and more complex integration tasks (research question 4).

Discussion

The goal of this study was to quantitatively and qualitatively describe the nature of students’ connection formation across texts, when asked to complete two types of multiple text tasks. In particular, by explicitly instructing students to engage in cross-textual connection formation during processing we were able to both elicit a fairly large number of instances of integrative reasoning from students and to analyze these in a systematic fashion. In this discussion, we expand on those elements of students’ integrative reasoning that we consider to be both most important and to contribute most to prior work on students’ learning from multiple texts.

Students’ performance across integration tasks

First, at least proportionately speaking, students were considerably more successful in relation formation in response to the elemental task as compared to the complex task. As described in the methods, this may be the case for a variety of reasons. To start, the elemental task asked students to draw connections across fewer texts (i.e., three) than did the complex task, which asked students to reason relationally across eight different texts. Moreover, the elemental task, while not including texts that made explicit or direct reference to one another, nevertheless, featured texts that were purposefully constructed to relate to one another in a variety of systematic ways.

The complex task, as a contrast, included texts that were more difficult to relate to one another for at least three reasons. First, the texts themselves were more complex than those introduced in the elemental task. Although forwarding arguments for and against the legalization of sex work, each text included a number of sub-arguments and pieces of evidence that may have been more difficult for students to map and connect across texts. Moreover, the claims or arguments provided in texts were of a more complementary nature; in contrast to arguments introduced in the elemental task that directly conflicted with one another in explaining why Stonehenge was constructed. For instance, arguments introduced across the sex work texts required students to connect the fact that making sex work illegal served to criminalize and stigmatize sexual behavior, while at the same time making certain sexual crimes easier to uncover and prosecute. Although these arguments represent conflicting points of view for and against legalization and both offer legal analyses, these represent a much more subtle and nuanced contrast than that introduced in the elemental multiple text task. Finally, the sex work texts were attributed to authors of quite varied backgrounds (e.g., academics, former sex workers now activists), potentially making contextual integration more of a challenge. In sum, the complex multiple text task presented students with a number of challenges in terms of forming evidentiary, thematic, and contextual relations across texts, potentially resulting in the more limited number of connections identified in association with this task, in proportion to the total number of connections able to be formed. In juxtaposing these two tasks we identify both elements that may facilitate students’ formation of connections across texts (e.g., fewer texts, parallel text structures) and therefore may be used to scaffold integrative reasoning; and those elements (e.g., more texts, representing varied perspectives, complex argument structures) that may make integrative reasoning a formidable challenge for learners. In considering these latter task elements, commonly occurring when students try to understand complex issues in the real world, we hope to provide insight into those aspects of multiple text integration that may require specific instruction and support [e.g., introducing students to a perspectives pallet to help them conceptualize various points of view during argumentative writing (Kiili et al. 2016)].

Level of integration

The key goal of this study was to articulate, on a cognitive level, the types of connections students form across texts, when making a deliberate effort to do so (i.e., when engaging in top-down integration, Kurby et al. 2005). In particular, students’ verbal reports were examined for evidence of each of the four levels of integrative reasoning identified by List & Alexander (2019), with these levels differing in their degree of specificity, abstraction, elaboration, and quality, overall. We discuss each of the identified levels, in turn, to describe the features that contribute to learners’ more quality and sophisticated integrative reasoning. We do this both to contribute to prior work aiming to understand students’ process of connection formation across texts (e.g., Anmarkrud et al. 2014; Wolfe and Goldman 2005) and to support the development of interventions aimed at fostering students’ integration of multiple texts. That is, in expounding on each level of integrative reasoning, in detail, we hope to provide insights into how expert models (e.g., teacher scripts, intelligent tutoring prompts), scaffolds (e.g., templates), and feedback schemes (e.g., rubrics) may be developed. We hope that such efforts help students not only identify more connections across texts but also form more specific and elaborated representations of cross-textual links. With these aims in mind, we consider each level of integrative reasoning introduced in this study, in greater detail.

The first level, relational identification, referred just to students’ identification of semantic overlap across texts. This was demonstrated in responses such as, “in both they talk about finding smaller blue stones,” from the elemental task, and “immigrants and minorities stuff go together, I think,” reported during the complex multiple text task. In both cases, while students were noting the semantic overlap across texts they were not explaining the meaning or purpose of this overlap (e.g., why blue stone chips were discussed in each text), resulting in a superficial connection formed, or one that did not aid students in constructing an overall representation or conceptualization of the common issue or topic discussed across texts.

Level 2, or separate representation, was reflected in cross-textual connections that, in addition to identifying the semantic overlap across texts, further explicitly stated or mapped which content, within each text, was related. An example of this type of connection formation was reflected in responses like: “Then on the first one it says the other reason that it was built was for healing. Then the second one it says that it not only attracted people who were unwell, but also healers who could aid them,” from the elemental task and “The eighth article talks about the use of drugs and alcohol to get through each night. And the seventh article touches upon how prostitution is associated with drug use and alcohol abuse,” from the complex multiple text task. In both cases, responses such as these informed the semantic overlap identified in the first level by further explaining what information, within each text, was being linked across texts. That is, while level 1 focused on the semantic overlap between texts in a fairly isolated or decontextualized fashion, a separate representations approach to integration was marked by a richer representation of linked information, appearing within each text, even if no connections were explicitly drawn. At the same time, these responses were not classified as reflecting simultaneous relations as students were not generating a novel, thematic connection across texts, rather these connections were formulated based only on the semantic overlap between texts.

The third level of integration, as identified by List & Alexander (2019), reflects simultaneous relation or the labeling or specification of an over-arching connection across texts. This level was demonstrated when students formed connections like: “,” and “So the first sentence of the fifth article says that, ‘Prostitution is a cycle of inequality.’ And that connects with the second article saying that, ‘Prostitution is oppression.’ So I’m gonna (sic.) highlight that green for lack of free will, exploitation, oppression,” from the elemental and complex tasks, respectively. As demonstrated by these examples, the specification of a simultaneous relation requires both that students separately represent the linked information occurring within each text and that they classify or generate a cross-textual link among them. In contrast to prior examples where the association that students were creating had to be inferred by the reader, in these examples the connection that students were forming was explicitly identified. In the first example, this connection was the importance of Stonehenge as a site, inferred from information presented across texts, while in the second example, the student identified the themes of ‘free will, exploitation, and oppression’. This, third level may be said to be the first true instance of integration as it is the first to require students to themselves form a novel link or super-ordinal relation across texts, one not explicitly appearing in the texts encountered.

The fourth and final level of integration, relational elaboration, corresponded to students developing a novel connection across texts and further explaining or interpreting the connection formed. This level of integration was reflected in responses to the elemental task like, “Then I’m highlighting ‘evidence for his theory comes from archeological excavations and geographic analysis,’ because in the first one it’s talking about how the original construction of Stonehenge was brought from over 200 miles away. I’m pretty sure geographic analysis is just studying the location of something, so these are both talking about location of Stonehenge,” and responses to the complex task, such as:

“Then in Prostitution is Female Oppression, it says, ‘At its core, prostitution is violence against women. Safe prostitution is an oxymoron.’ Then in Legalizing Sex Work Promotes Public Health, it says, ‘Prostitution is a reality. Eliminating prostitution is unrealistic public policy proposal. Our only choice is to make sex work as safe as possible, as we do with any other profession.’ The one is saying it’s impossible to have safe sex work, and the other one is saying it’s impossible to eliminate it, so we should make it as safe as possible.”

In both of these examples, students not only generated a novel, super-ordinal connection across texts but also explained and substantiated the connection formed. Indeed, this last level of integration was considered to reflect the ultimate degree of connection formation needed for students to fully and holistically understand multiple texts. That is, the connection formation reflected in this level of integration, was not only fully formed but also helpful to students in understanding the texts provided in a deeper and more meaningful way.

Throughout these four levels, students’ relative sophistication in integrative reasoning is exemplified and its function in supporting multiple text comprehension is demonstrated. As such, given their progressive nature, we consider these four levels and their articulation to be a viable framework to use in developing and assessing students’ integrative reasoning when learning from multiple texts.

Limitations

Despite the strengths of this study a number of limitations must be acknowledged. First and foremost, a number of limitations come from the distinctive nature of the elemental and complex integration tasks used in the study. Although these were not intended to be compared directly, the number of differences across these two tasks (e.g., texts’ number, structure, source attribution) renders the differences in integration identified across them impossible to systematically specify. More generally, the types of connections that students identified, across tasks, may be considered to be somewhat idiosyncratic to the study materials used in this study and to the particular students who participated. We tried to mitigate this limitation by using a within-subjects design and asking students to complete two tasks, varying along a variety of dimensions. We, nevertheless, must acknowledge that if students were presented with texts describing causes and effects (Wiley et al. 2009) or problems and solutions List, Du, Wang, & Lee (2019), the types of connections that students identified, as well as their specificity and the level of integrative reasoning that these reflected, would likely differ from those identified in this study. As such, future work should investigate students’ connection formation across texts related to one another in a variety of ways.

Second, the methods used in this study present a number of limitations. To start, in this study, rather than asking students to think-aloud, students were explicitly directed to identify connections across texts. On the one hand, this methodological decision was made to allow students to purposefully form and elaborate cross-textual connections (i.e., in a top-down fashion, Kurby et al. 2005) and to increase students’ engagement in integration, overall. On the other hand, this leaves questions regarding the extent to which the connections that students formed were those that they would have spontaneously identified during reading, if they had not been directed to do so. As an added point, the instructions provided to students prior to reading specifically directed them to explain the connections that they formed across texts. This may have resulted in students providing more elaboration and more level 4 instances of integration than they would have otherwise. Nevertheless, even with these directions, students were still found to report cross-textual connections across all four levels of integration and to report higher-level connnections (e.g., Level 3, Level 4) to a fairly limited extent.

Moreover, while this study contributes valuable examples of the different levels of integration that students may demonstrate, when reasoning about texts, the extent to which these levels are indeed progressive in sophistication requires further exploration in a number of ways. For instance, the nature of students’ ordered progression from level 1 to level 4 requires further examination, as does the extent to which level 1 vis-à-vis level 4 truly reflect differences in reasoning, rather than differences only in students’ degree of articulation and elaboration when verbally reporting the cross-textual relations that they formed.

Finally, the nature of the association between cross-textual connection formation during multiple text processing and task performance needs to be better understood. In this study, focused on processing, the establishment of a connection between processing and performance was compromised by both the small sample and the low reliability of the evidence sort task. Nevertheless, analyzing the connection between quality of integrative reasoning and multiple text task performance would offer important insights into the hierarchical and comparatively more adaptive nature of students’ connection formation across levels of integration.

Conclusion and implications

This study contributes to the emergent literature on multiple text integration in at least three key ways. First, by explicitly instructing students to engage in integration, it captured many more integrative relations across texts than have been reported in prior work. Second, it adapted List & Alexander (2019) proposed cognitive model of integration into a framework that could be used to classify the sophistication or elaboration of the cross-textual relations that students form when reasoning about multiple texts. Finally, this study presented students with two different types of tasks intended to foster integration, one more elemental and the other more complex. While differing in a variety of ways, principally these tasks were intended to represent more explicit (i.e., elemental task) vis-à-vis more ambiguous (i.e., complex task) integration scenarios, potentially reflective of the range of multiple text tasks that students may be asked to complete across academic contexts.

Connecting these varied contributions suggests both that students are challenged by integration tasks that are more explicit and more ambiguous in nature and that helping students progress from recognizing the semantic overlap across texts, to identifying and elaborating super-ordinal relations may be a promising approach to intervention development. That is, analyses in this study offer initial insights into how students may be supported to engage in integration during multiple text use. This includes teaching students about the types of connections that they can form (i.e., evidentiary, thematic, and contextual) as well as introducing the different levels of integration and their various characteristics. In particular, the four levels identified in this study can be used as an instructional sequence for developing students’ more low-level or bottom-up connections, formed based on the semantic overlap between texts (i.e., Level 1 connections), into higher-level cross-textual connections. This likely requires prompting students’ reasoning to achieve higher levels of integration sophistication and explaining to students the purpose of forming higher-level or more elaborated connections across texts. Moreover, these four levels of integrative reasoning may be used to assess the quality of students’ cross-textual connection formation and to provide students with feedback. Nevertheless, these all represent directions for future work that are contingent on better understanding students’ integrative reasoning during multiple text use and its relationship with task performance.