Introduction

Two areas of great current interest among educational psychologists and reading researchers concern epistemic beliefs, that is, beliefs about knowledge and knowing, and multiple-documents comprehension, that is, the building of a coherent, meaningful representation of a situation or issue discussed across documents. With foundational work in each area taking place in the early 1990s (Schommer 1990; Wineburg 1991), there were only a few scattered attempts to bring them together before the turn of the century (Jacobson and Spiro 1995; Rukavina and Daneman 1996). In the period when research on epistemic beliefs and multiple-documents comprehension essentially traveled on parallel paths, however, links were established between epistemic beliefs and readers’ comprehension of single texts. For example, within the multidimensional frameworks of epistemic beliefs proposed by Schommer (1990) and Hofer and Pintrich (1997), respectively, both describing dimensions concerning the certainty and simplicity of knowledge, it was found that beliefs in certain rather than tentative knowledge negatively predicted interpretation and comprehension of text (Kardash and Scholes 1996; Schommer 1990; Schommer and Dunnell 1997; Schommer and Walker 1995), as did beliefs in simple rather than complex knowledge (Buehl and Alexander 2005; Schommer et al. 1992; Schommer and Walker 1995; Schraw et al. 2002). In addition, two dimensions involving beliefs about knowing were described in those frameworks—beliefs concerning the source of knowledge (termed “omniscient authority” by Schommer) and beliefs concerning the justification for knowing (described by Hofer and Pintrich but not included in Schommer’s belief system). However, empirical evidence linking those two dimensions to single-text comprehension is essentially lacking, at least with respect to expository texts (see Bråten et al. 2008; Strømsø and Bråten 2009). There is some evidence that viewing the reader rather than the author as the source of knowledge may be positively related to the interpretation of narrative text, though (Schraw 2000; Mason et al. 2006). Please note that the term “source” is used quite broadly in the present paper, not only in referring to diverse printed (e.g., a printed book) or digital (e.g., an online magazine article) documents that may provide individuals with information on some topic, but also in referring to origins of knowledge and knowing that may be internal (e.g., personal opinion or experience) or external (e.g., an external authority) to individuals. With respect to “beliefs”, we use this term in referring to what individuals accept as or want to be true that, more or less consciously, may prime or guide particular actions (cf., Murphy and Mason 2006).

Not surprisingly, the emergence of theory and research on multiple-documents comprehension coincided with the spurt-like development of the information society, with contemporary literacy activities, both in and out of school, fundamentally involving intertextual practices such as searching for, selecting, processing, and integrating information from multiple documents, printed as well as digital (Goldman 2004; Goldman et al. 2011; Rouet 2006). With the explosive growth of digital media content in the last decades, determining the usefulness (i.e., relevance) and credibility (i.e., trustworthiness) of sources has also become a major challenge, in particular for young people (Metzger and Flanagin 2008). From the beginning, systematic research on multiple-documents comprehension was concerned with the importance of prior knowledge, strategies, and tasks to the construction of an integrated understanding from conflicting information located in multiple documents. Thus, the important role played by both domain and documentary expertise (i.e., experience and skill in handling multiple documents) was shown by several early studies (Rouet et al. 1996, 1997; VanSledright and Kelly 1998; Wineburg 1991), highlighting that prior knowledge allows for bridging inferences that build links and coherence across documents. Moreover, with respect to an adaptive strategic approach, researchers focused on the importance of activities such as source evaluation, corroboration, and contextualization (Britt and Aglinskas 2002; Rouet et al. 1996; Stahl et al. 1996; Strømsø et al. 2003; Wineburg 1991; Wolfe and Goldman 2005), and with respect to tasks, on the potentially facilitative effect of argument-centered writing tasks (Wiley and Voss 1996, 1999).

Although none of the studies of multiple-documents reading cited above empirically explored the relationship between epistemic beliefs and multiple-documents comprehension, some of the authors argued convincingly for the importance of examining those variables in conjunction. For example, Stahl et al. (1996) argued that to build an integrated understanding from conflicting information located in multiple documents, readers need to develop an epistemic stance involving that knowledge is viewed as constructed through both rational processes and the melding of information from different perspectives. In the same vein, Wolfe and Goldman (2005) posited that to process and reason from multiple information sources, students need to realize that interpretations of events are often complex, involving multiple perspectives and with no single document or pair of documents likely to provide absolute truths. In the last decade, these notions have gained considerable empirical support, and, as recently reviewed by Bråten et al. (2011a), all four epistemic belief dimensions figuring in the influential Hofer and Pintrich (1997) conceptualization (i.e., beliefs concerning the certainty of knowledge, the simplicity of knowledge, the source of knowledge, and the justification for knowing) have now been linked to multiple-documents comprehension. In brief, viewing knowledge as tentative rather than certain, complex rather than simple, originating in expert authors rather than the reader, and justified by rules of inquiry and cross-checking of knowledge sources rather than own opinion and experience have been shown to uniquely predict students’ ability to synthesize information from expository documents expressing diverse and even contradictory viewpoints on a particular topic (for review, see Bråten et al. 2011a).

One limitation with existing research on epistemic beliefs in relation to multiple-documents comprehension is that it has almost exclusively been conducted within the multidimensional framework of Hofer and Pintrich (1997). However, this landmark framework is currently challenged by conceptualizations drawing more heavily on philosophical epistemology, specifically by the multidimensional framework proposed by Greene et al. (2008) and the multicomponential framework proposed by Chinn et al. (2011). Whereas Chinn et al. primarily expand the Hofer and Pintrich framework by adding new components concerning cognitions about epistemic aims (e.g., knowledge) and epistemic values (e.g., the worth of knowledge), epistemic virtues (e.g., dispositions that aid in attaining knowledge) and vices (e.g., dispositions that hinder the attainment of knowledge), and reliable and unreliable processes for achieving epistemic aims (e.g., argumentation as a way to attain knowledge), Greene et al. argue that because philosophical epistemology centers on how claims can be justified as knowledge, only one of the four dimensions described by Hofer and Pintrich, concerning justification for knowing, rightly deserves to be labeled epistemic. According to Greene et al., the nature of knowledge dimensions, certainty and simplicity of knowledge, should be combined and rather considered “ontological” because they concern people’s views on reality in particular domains. At the same time, they posited that justification for knowing should be elaborated and differentiated into more than one dimension in accordance with philosophical epistemology, which identifies a number of different sources, both internal and external to the individual, that can be legitimately used to justify knowledge claims, especially highlighting separate dimensions concerning personal justification and justification by authority in their proposed model. Thus, in contrast to Hofer and Pintrich, who conceived of different beliefs concerning justification for knowing as located on a single continuum or dimension, ranging from justification of knowledge claims through observation, authority, or what feels right at lower levels to the use of rules of inquiry and the evaluation and integration of different perspectives at higher levels, Greene et al. presumed that different forms of justification constitute separate dimensions. Please note that it also follows from a multidimensional conceptualization of justification for knowing beliefs that different dimensions may operate more or less independently, meaning that a person may be high on one dimension and, at the same time, high, moderate, or low on another dimension. Likewise, Chinn et al. included several distinct types of justification in their framework, highlighting the need to study justification at a finer grain size across different contexts.

In the current study, we built on the multidimensional framework proposed by Greene et al. (2008) and conceptualized epistemic beliefs in terms of several dimensions of justification for knowing, with those dimensions differentiated by the types of sources that students draw on in their effort to justify knowledge claims. By doing this, we also subsumed the source of knowledge dimension figuring in the Hofer and Pintrich (1997) framework under justification for knowing. Specifically, in addition to including the person and external authority as potential sources of justification, we assumed that students use multiple external sources to evaluate knowledge claims. With respect to level of specificity, Greene et al. (2008) suggested that epistemic cognition varies at a level between domain generality and domain specificity, proposing differentiation between well-structured (e.g., mathematics and physics) and ill-structured (e.g., history and literature) domains. However, because much evidence indicates that students’ epistemic beliefs differ between specific academic domains or disciplines (for reviews, see Buehl and Alexander 2001; Limon 2006; Muis et al. 2006), and because the Greene et al. conceptualization also seems applicable to a domain-specific level of epistemic beliefs, we chose to target justification beliefs concerning the domain of science in the current study.Footnote 1 We believe that examining this alternative conceptualization of epistemic belief in relation to multiple-documents comprehension may not only benefit the area of epistemic beliefs but also provide new understanding of what it takes to be a competent reader in the information age. However, before we specify the questions and hypotheses that guided our empirical investigation, we review prior work on justification beliefs and multiple-documents comprehension and explain why we found it necessary to control for prior knowledge in our study of relationships between those constructs.

Justification beliefs and the comprehension of multiple documents

As reviewed by Buehl (2008), the justification for knowing dimension, as described by Hofer and Pintrich (1997), has seldom been identified in factor analytic studies. Moreover, quantitative evidence linking this dimension to any kind of performance measure is still sparse. Notable exceptions are a series of studies by Mason et al. (2010a, b, 2011), which indicated that across educational levels, students likely to consider the justification of knowledge claims in terms of scientific evidence were also likely to learn more from multiple web-based sources. It can be argued, however, that participants in those studies were assigned a search task rather than a reading task and, more importantly, that the dependent measures targeted the learning of correct information units rather than integrated multiple-documents comprehension.

More directly related to multiple-documents comprehension, Bråten and Strømsø (2010b; Strømsø & Bråten 2009) recently showed that some justification beliefs may, indeed, be linked to the comprehension of multiple conflicting documents. In those studies, the justification dimension reflected a continuum where high scores represented the belief that knowledge claims should be based on rules of inquiry and the evaluation and integration of multiple knowledge sources, while low scores represented the belief that knowledge claims can be justified through own opinion, firsthand experience, or common sense. It should also be noted that justification beliefs were measured with respect to a particular topic (i.e., justification for knowing about climate change), and that students read multiple conflicting documents dealing with the same topic.

In the Strømsø and Bråten (2009) study, which used a sample of secondary school students, the contribution of justification beliefs was found to override the contribution of topic knowledge as well as topic interest to both within- and cross-text comprehension. Also, in the Bråten and Strømsø (2010b) study, using a sample of relatively experienced law students, it was found that a belief in justification through rules of inquiry and cross-checking of knowledge sources predicted students’ meaning construction independent of their prior knowledge about the topic. In a previous study with first-year education undergraduates, however, Strømsø et al. (2008) did not find any unique relationship between justification beliefs, measured with the same topic-specific questionnaire, and meaning construction, neither within nor across documents. Although findings are not unequivocal, then, even the multiple-documents comprehension of students in preundergraduate education may seem to be facilitated by beliefs in the need to critically examine, evaluate, and integrate multiple information sources when encountering knowledge claims.

Of note is that the empirical work cited above builds on a somewhat broader conceptualization of the justification for knowing dimension than what is found in the framework of Hofer and Pintrich (1997) (see above). Thus, as part of this dimension, Bråten and colleagues (Bråten and Strømsø 2010b; Strømsø and Bråten 2009; Strømsø et al. 2008) counted beliefs in justification through reason (i.e., critical thinking), prior domain knowledge, scientific inquiry, and cross-checking of sources at one end of the continuum and beliefs in justification through own opinion, first-hand experience, and common sense at the other end of the continuum. Still, these authors considered different forms of justification to be located on a single continuum or dimension, as did Hofer and Pintrich, rather than constituting different dimensions, as proposed by Greene et al. (2008).

In a think-aloud study investigating education undergraduates’ epistemic cognition during the reading of multiple conflicting documents on the topic of cell phone radiation and potential health risks, however, Ferguson et al. (2012a) found that epistemic cognition regarding the topic was represented by three different dimensions concerning justification for knowing. Specifically, in addition to personal justification and justification by authority, also described by Greene et al. (2008), a third dimension involving that students considered which claims to believe on the basis of cross-checking, comparing, and corroborating across several sources of information was identified. In a later, factor-analytic study, Ferguson et al. (2012b) demonstrated that the same three dimensions, that is, personal justification, justification by authority, and justification by multiple sources, could be identified at a science-specific level in questionnaire data from secondary school students. Moreover, the latter study suggested that working with multiple conflicting documents on a science issue might decrease students’ beliefs in personal means of justification and increase their beliefs in justification by multiple sources. None of the studies by Ferguson et al. (2012a, b) examined relations between the three justification dimensions and multiple-documents comprehension. This was done by Bråten et al. (2012), however, who identified the same three dimensions of justification in think-aloud protocols based on education undergraduates’ reading of six documents presenting different perspectives on cell phone radiation and health risks. After controlling for topic knowledge, these authors found that justification by multiple sources uniquely predicted performance on argumentative essays that students wrote after reading the documents. More specifically, students trying to justify knowledge claims by corroborating across several sources of information were also more likely to include explicit source citations, link sources and contents, and display more integrated argumentation in their essays, with this indicating that they were more likely to construct documents models in the sense of Britt and colleagues (Britt et al. 1999; Britt and Rouet 2012; Perfetti et al. 1999).

In summary, there is plenty of room for increased clarity regarding students’ different means of justifying knowledge claims and their potential importance to students’ comprehension of multiple documents. Hopefully, by addressing not only to what extent students rely on the different means of justification, but also how those means differentially relate to multiple-documents comprehension, the present study may make that room somewhat smaller.

Controlling for prior knowledge

Because our main focus was on the unique predictability of justification beliefs for the comprehension of multiple documents, we also wanted to remove variance associated with prior knowledge before examining the potentially unique contributions of different dimensions of justification beliefs. Theory and research within the single-text paradigm have long established that students’ comprehension performance is influenced by the knowledge they already possess, with an abundance of evidence supporting the importance of prior knowledge for single-text comprehension (for reviews, see Anderson 2004; Fox 2009; Kintsch 1998). Relevant for multiple-documents comprehension, in particular, the cognitive flexibility theory of Spiro et al. (e.g., 1991, 1994) suggests that exploring contrasting perspectives located in multiple sources will be more beneficial at relatively advanced than at introductory levels of domain knowledge. Correspondingly, research conducted in the 1990s (e.g., Rouet et al. 1996; Stahl et al. 1996; Wineburg 1991) indicated that students with limited prior knowledge may have difficulties integrating information across multiple historical source documents. More recently, Bråten and colleagues (Bråten and Strømsø 2006, 2010a, b; Gil et al. 2010; Strømsø and Bråten 2009) have shown that students’ prior knowledge about the topic of the documents is a predictor of their comprehension performance when reading multiple conflicting documents on a scientific issue. Other research outside the domain of history has also provided evidence that multiple-documents comprehension is associated with prior knowledge (e.g., Kobayashi 2009; Le Bigot and Rouet 2007; Moos and Azevedo 2008; Pieschl et al. 2008). Presumably, prior knowledge contributes to comprehension performance in these studies because it facilitates bridging inferences that create interconnection and coherence in complex, divergent text materials.

Of note is that prior knowledge might be considered a predictor of not only multiple-documents comprehension but also of epistemic beliefs. It could thus be argued that epistemic beliefs are largely an outcome of prior knowledge (cf., Bromme et al. 2008). This is another reason why we were particularly interested in whether beliefs about justification for knowing would override the contribution of prior knowledge about the topic of the documents to multiple-documents comprehension.

The present study

On this conceptual and empirical backdrop, we set out to examine adolescents’ beliefs concerning the justification of knowledge claims in the context of reading multiple conflicting documents on the social-scientific issue of sun exposure and health. By doing this, we extended prior research in several ways. First, we extended prior research on the role played by epistemic beliefs in students’ multiple-documents comprehension by focusing on an age group that has hardly been included in prior work, yet an age group for which such complex reading tasks seem developmentally appropriate (Alexander and Fox 2011). Second, we extended prior research by investigating different dimensions of domain-specific justification beliefs as predictors of multiple-documents comprehension. Third, while prior research on justification beliefs and multiple-documents comprehension has mainly used yes/no inference verification tasks to assess intertextual comprehension of multiple documents, this study used short-essay integrative questions to measure readers’ ability to integrate content from different perspectives.

Specifically, we addressed two questions. First, we asked whether beliefs in personal justification, justification by authority, and justification by multiple sources with respect to knowledge claims in natural science differed in strength among the participating adolescent readers. Given that students at different educational levels have been shown to put much trust in external authority, such as the textbook (Bråtenet et al. 2011b; Paxton 2002) and the teacher (Hofer 2004), we hypothesized that participants would most strongly believe that knowledge claims in natural science should be justified by authority. Moreover, we hypothesized that participants would be least likely to believe in personal justification of knowledge claims in natural science, consistent with findings that students do not seem to rely much on personal means of justification in hard, well-structured domains (for review, see Muis et al. 2006).

Second, we asked whether science-specific beliefs in personal justification, justification by authority, and justification by multiple sources, respectively, differentially and uniquely predicted the comprehension of multiple conflicting documents on a science issue. Regarding the second question, we expected that a belief in personal justification of knowledge claims would be a unique, negative predictor of multiple-documents comprehension. This expectation was based on prior research on multiple-documents comprehension (Bråten et al. 2008; Strømsø et al. 2008) indicating that relying on personal opinion and judgment may be maladaptive because readers concentrate too much on the subjective and fail to figure out precisely what the authors and documents actually say. Moreover, we expected that a belief in justification by multiple sources would be a unique, positive predictor of multiple-documents comprehension. This expectation was based on the recent study of Bråten et al. (2012), where topic-specific justification by multiple sources assessed through think-aloud data collected during the reading of multiple conflicting documents on another science topic uniquely predicted undergraduates’ integrated argumentation assessed through essay performance. Also, this expectation seems consistent with other research on multiple-documents comprehension indicating that corroboration of information across multiple documents may facilitate the building of integrated comprehension (for review, see Afflerbach and Cho 2009). With respect to justification by authority, previous findings are mixed, with some work (Bråten et al. 2008; Strømsø et al. 2008) indicating that reliance on expert authors may be positively related to multiple-documents comprehension, and other work (Bråten et al. 2012) indicating no relationship between those constructs. In the present study, we therefore decided to explore this issue further without forwarding any specific, directional hypothesis. On the one hand, justification by authority may be linked to better multiple-documents comprehension because it makes readers accept the authority of informed authors and focus on their intended messages; on the other hand, readers more likely to believe in justification by authority may be confused and frustrated when trying to construct an integrated understanding from multiple documents where several experts represent opposing views on an issue.

Method

Participants

Participants were 65 10th graders who volunteered from eight classes from four public lower secondary schools in and around a city in southeast Norway. The sample included 39 girls and 25 boys (one did not report on gender) with an overall mean age of 14.9 (SD = 0.24). Participation was anonymous and remuneration entailed entry into a prize draw for a gift card (∼40 USD) for a music store. Sixty-one participants were native Norwegian speakers and only four came from families where neither parent spoke Norwegian as their first language. In an international perspective, the sample was relatively homogeneous (i.e., middle class) in regard to socioeconomic status.

Materials

Topic knowledge measure

To assess topic knowledge, we developed a multiple-choice test composed of 20 items. The content of the items referred to concepts and information central to the issue of sun exposure and health that were discussed in the five documents that they read (see below). In creating the measure, the first and the third author independently selected key concepts and information from the documents and together wrote items to cover the concepts and information agreed upon by both test constructors (e.g., ultraviolet radiation, vitamin D, skin cancer, and sun protection). Taken together, the 20 items assessed both conceptual understanding and factual knowledge regarding the issue. A preliminary version of the topic knowledge measure was reviewed by a professor of medical biochemistry at the University of Oslo who was not part of the project, with this resulting in only minor modifications to the response alternatives of a few items. Sample items from the topic knowledge measure are displayed in Appendix A.

Whereas the national curriculum in science for tenth grade states that students should gain knowledge about “the body, diseases, and various ways to protect against them”, “radiation” is not taught until the first year of upper secondary school (i.e., Grade 11; Norwegian Ministry of Education and Research 2006). Still, participants’ topic knowledge could be expected to vary considerably because those interested in the issue might have acquired such knowledge by other means. Participants’ topic knowledge score was the number of correct responses out of the 20 items. The reliability (Kuder–Richardson 20) for scores on the measure was 0.64.

Justification belief measures

To assess beliefs about justification for knowing, we used 14 items from the Justification for Knowing Questionnaire (JFK-Q) previously described by Ferguson et al. (2012b). This is an 18-item questionnaire based on the multidimensional conceptualization of justification for knowing of Greene et al. (2008), with items written to capture justification of knowledge claims by means of different types of sources. Specifically, the JFK-Q is intended to capture separate dimensions concerning personal justification, justification by authority, and justification by multiple sources, with all items pertaining to the domain of natural science. Of the six items included to assess beliefs concerning personal justification for knowing, four items are adapted from the personal justification items included in Greene et al.’s (2010) Epistemic and Ontological Cognition Questionnaire (EOCQ). Likewise, of the six items included to assess beliefs concerning justification by authority, four items were adapted from the justification by authority items included in the EOCQ (Greene et al. 2010). Exploratory and confirmatory factor analyses reported in prior research using a larger sample of Norwegian 10th graders (Ferguson et al. 2012b) indicate that the three dimensions of justification for knowing beliefs indeed seem to underlie students’ scores on the JFK-Q. Because the sample size in the present study was too small to conduct factor analysis (Tabachnick and Fiddell 2007), we included in each of our three measures the items that fell on that particular dimension in the factor-analytic study of Ferguson et al. (2012b). However, given that the latent structure in the data was quite clear when examined by both exploratory and confirmatory factor analyses (Ferguson et al. 2012b), we consider our justification belief categorization to be empirically (as well as theoretically) justifiable.

Thus, beliefs about personal justification for knowing were assessed with three items concerning the extent to which students considered it appropriate to evaluate knowledge claims in natural science on the basis of personal views or opinions (sample item: “What is a fact in natural science depends on one’s personal views”). The higher the scores on this measure, the more students can be assumed to believe that knowledge claims can be justified by appealing to subjective, internal means of justification. Beliefs about justification by authority were assessed with six items focusing on the reliability of statements or claims based on scientific research and conveyed by teachers, textbooks, and scientists (sample item: “When I read something about natural science that is based on scientific investigations, then I believe that it is correct”). The higher the scores on this measure, the more students can be assumed to believe that knowledge claims can be justified by appealing to an authoritative external source or evidence derived from scientific research. Finally, beliefs about justification by multiple sources were assessed with five items concerning the extent to which students found it necessary to cross-check and corroborate claims across several sources of information (“To detect incorrect claims in texts about natural science, it is important to check several information sources”). Higher scores on this measure can thus be assumed to represent stronger beliefs in the importance or necessity of justifying knowledge claims in natural science by checking multiple external sources for consistency. All items included in the three justification belief measures that we used are displayed in Appendix B.

Participants rated each item on a 10-point anchored scale (1 = disagree completely, 10 = agree completely). The reliability estimates (Cronbach’s α) for scores on the measures of personal justification, justification by authority, and justification by multiple sources, respectively, were 0.63, 0.81, and 0.72.

Documents

The five documents that we used presented different perspectives on sun exposure and health. We selected these documents because they represented different kinds of authentic source materials that students would typically encounter when seeking information about this unsettled scientific issue, and because the discussion of this issue, being highly topical for people in northerly regions where sunlight is scarce during winter and extremely attractive during summer, was likely to elicit engagement on part of the students. At the beginning of each document, source information was presented in the form of author’s name and credentials, publisher, document type, and date of publication.

The first document was a 382-word excerpt from a first year upper secondary science textbook. The document described different types of ultraviolet radiation in neutral, academic terms, concluding that more research is needed to clarify the health implications of such radiation. The second document was a 398-word popular science article from a university research magazine, citing recognized scientists who advocated more sun exposure for northerners because it is an essential source of vitamin D, also presenting research-based evidence that exposure to ultraviolet radiation, outdoor as well as indoor (i.e., sunbed), may actually protect against cancer, particularly in internal organs. The third document was a 393-word popular science article from an online research magazine published by a group of educational institutions. This document presented an interview with a professor who explained and provided evidence for the causal relation between ultraviolet radiation and skin cancer, claiming that tanning is a dangerous way to obtain vitamin D and therefore suggesting cod-liver oil or supplements for those who lack vitamin D. The fourth document was a 323-word article from a Norwegian newspaper referring to a large-scale longitudinal study indicating that sunrays may protect against all types of cancer through the production of vitamin D, also recommending at least 30 min of daily sun exposure. Finally, the fifth document was a 375-word public information text published by the Norwegian Cancer Association, describing different types of skin cancer that may be caused by ultraviolet radiation and suggesting ways of reducing the risk of developing skin cancer (e.g., avoiding sunbeds and staying out of the sun between 10:00 a.m. and 4 p.m.).

Thus, apart from the more neutral science textbook excerpt, the four other documents contained partly conflicting information, with two documents (3 and 5) arguing that ultraviolet radiation may cause skin cancer and two documents (2 and 4) arguing that ultraviolet radiation may protect against cancer through the production of vitamin D. Table 1 provides an overview of the five documents that were used in this study.

Table 1 Overview of the five documents
Full size table

As an indication of difficulty, we used the formula of Björnsson (1968) to compute readability scores for each of the documents. This formula, which is based on word length and sentence length, yields readability scores ranging from about 20 (very easy) to about 60 (very difficult). Vinje (1982) reported that textbooks used in Norwegian high schools had a readability score of approximately 42 and that public information texts from the Norwegian government had a readability score of 45. Table 1 shows that the readability scores of the five documents ranged from 37 to 46 (M = 41.6, SD = 3.9), suggesting that they represented a suitable challenge for our participants.

Multiple-documents comprehension measure

To assess multiple-documents comprehension, we asked participants to answer three open-ended, short-essay questions in writing. The questions were modeled on the integrative short-essay questions used by Rukavina and Daneman (1996) to assess students’ understanding of a controversial scientific issue. Following those authors, we considered our first question to indirectly require participants to integrate perspectives across documents or, at least, to consider each perspective’s claim and reasons, in order to answer the question. Also following Rukavina and Daneman, we considered our second and third question to directly require participants to pit perspectives against each other, measuring how well they could reason about the issue in terms of the claims and reasons presented in the documents.Footnote 2

The indirect integrative question was “Explain the relationship between sun exposure, health, and illness.” Responses were coded using a 0–4 coding scheme designed to assess how well participants explained the issue and integrated the different perspectives discussed in the documents. The coding scheme for this question was as follows: level 0 = no response or irrelevant information; level 1 = merely mentioning one or two of the perspectives (i.e., sun exposure is harmful and sun exposure is healthy, respectively) with no explanation or reason for any of them; level 2 = mentioning one or two of the perspectives and providing explanation or reason (e.g., that sun exposure may cause skin cancer and that sun exposure may increase vitamin D production, respectively) for one or both; level 3 = mentioning one or both of the perspectives and providing elaborate explanation or reason (e.g., by specifying different types of cancer that different types of ultraviolet radiation may cause or protect against) for one or both; and level 4 = mentioning the two perspectives and providing elaborate explanation or reason for one or both as well as relating the two perspectives to each other (e.g., comparing, contrasting, or trying to reconcile the two perspectives). First, the first, third, and fourth author collaboratively scored the responses of five participants. Then, a random selection of 13 participants’ responses (i.e., 20 %) to the first question was independently scored by the third and the fourth author, resulting in 80 % agreement and with all disagreements solved through discussion. The responses of the rest of the participants were scored by the fourth author alone.

The first direct integrative question was “There are different views on the relationship between sun exposure, health, and illness. Describe important differences between these views.” The coding scheme for this question was as follows: level 0 = mentioning no perspective or providing irrelevant information; level 1 = merely mentioning one perspective with no explanation or reason; level 2 = mentioning one perspective and providing explanation or reason for this; level 3 = mentioning the two perspectives without providing explanation or reason for any of them; level 4 = mentioning the two perspectives and providing explanation or reason for one of them; and level 5 = mentioning the two perspectives and providing explanation or reason for both of them. On this question, all four authors collaboratively scored 17 participants’ responses (i.e., 26 %), with all disagreements solved through thorough discussion. The responses of the remaining participants were scored by the fourth author alone.

The second direct integrative question was “Could more than one view on the relationship between sun exposure, health, and illness be correct? Yes or no? If no, why not? If yes, why?” Basically, the scoring procedure occurred in two steps. In step 1, we recorded whether participants recognized that the two perspectives were not mutually exclusive or might be reconciled (i.e., whether they answered “yes” or “no”). In step 2, we assessed to what extent participants could explain and reconcile the two perspectives (i.e., when they answered “yes”) and to what extent they could select one of the perspectives and provide explanation or reason for that (i.e., when they answered “no”). Thus, participants who merely answered “yes” to the first question were assigned to level 1. To obtain a higher level they, in addition, had to mention the two perspectives without explanation or reason (level 2), mention the two perspectives and providing explanation or reason for one or both (level 3), mention the two perspectives and provide elaborate explanation or reason for one or both (level 4), or mention the two perspectives and provide elaborate explanation or reason for one or both as well as relating the two perspectives to each other by explaining how both may be correct (i.e., reconciliation; level 5). On the other hand, participants who merely answered “no” to the first question were assigned to level 0. To obtain a higher level they, in addition, had to select one of the perspectives (i.e., state a position on the issue) without explanation or reason (level 1), select one of the perspectives and provide explanation or reason for this (level 2), or select one perspective and provide elaborate explanation or reason for this (level 3). On this question, the first and the third author first scored 10 participants’ responses collaboratively. Then, the same two authors independently scored a random selection of 15 participants’ responses (i.e., 23 %), reaching an agreement of 82 % and solving all disagreements through discussion, before one of them scored the responses of the remaining participants.

The possible range of scores was thus 0–4 on the first question, 0–5 on the second question, 0–5 on the third question, and 0–14 on the entire measure. Only participants’ total scores on the entire measure were used in subsequent statistical analyses.

Procedure

Data for the study were collected in two separate sessions. In the first, 30-min session, the topic knowledge measure and the JFK-Q were group administered in that order in participants’ ordinary classrooms. These measures were paper and pencil measures containing a short written instruction that participants were told to read carefully, with the test leader ensuring that the instructions were understood. One week after the first session, each participant read the five documents about sun exposure and health and responded to the short-essay questions during a 60-min session that took place in a computer lab at the school. The documents were read on a computer with the Read&Answer 2.0 software (Vidal-Abarca et al. 2011). To ensure that participants were comfortable with the workings of this software before proceeding to the actual reading task, they were first guided through a 10-min training session, using different documents on an unrelated topic. Before starting on the actual reading task, they read the following instruction: “You shall now read five different texts on sun exposure and health. The texts are taken from the Internet. While reading, imagine that you are going to hold a presentation for the rest of the class about how sun exposure affects our health.” Participants were also told that they could read the documents in any order. In the Read&Answer application, each document was presented on a separate page and a simple interface allowed participants to navigate within and across documents. One whole page was visible at any given time, with all the text on the page masked except one segment currently selected by the participant. The participant could unmask and thereby read a segment of approximately 70 words by clicking on the segment, and when clicking on another segment in any location on the same page or on another page, the previously selected segment was remasked so that only one segment was legible at one time. The application thus permitted participants to jump between documents in any order and unmask a segment in any location on any page. Importantly, it also allowed us to check that readers actually accessed each segment of each text. Please note that experimental work by Vidal-Abarca et al. (2011) demonstrates that reading comprehension scores are not affected by the masking function of the Read&Answer.

After reading, participants closed the application and responded to the three short-essay questions on paper. Thus, they did not have access to the documents during comprehension assessment. Participants were not given access to the documents while working on the short-essay questions because we wanted them to respond on the basis of the mental representations constructed during reading rather than on the basis of searching for, locating, and copying information at the time of responding to the questions. The general instruction for the short-essay questions was: “Here are some questions concerning what you have just read. Answer the questions as fully as possible. Use the time you need for reflection when answering.” Participants were given two lined sheets of paper that were stapled together, with the general instruction and the first question printed at the top of the first sheet, and with the second and the third question printed at the top and in the middle of the second sheet, respectively. All data for the study were collected by the second and the fourth author together with two trained research assistants.

Results

Descriptive data (means, standard deviations, skewness values, and kurtosis values) for all measured variables are presented in Table 2. As can be seen, the skewness values ranged from −0.31 to 0.26 and the kurtosis values ranged from −0.86 to 0.10, indicating that all score distributions were approximately normal and, thus, appropriate for use in parametric statistical analyses.

Table 2 Descriptive statistics for measured variables
Full size table

We conducted a repeated-measures analysis of variance to address our first research question, concerning whether there were differences between participants’ scores on the three justification belief measures. In this analysis, we used the Huynh–Feldt correction to adjust the degrees of freedom because the homogeneity (sphericity) assumption was violated. Results indicated that participants did not endorse the different means of justifying knowledge claims in science to the same extent, with F(1.98, 124.50) = 49.80, p = 0.000, partial η² = 0.44. Follow-up paired-sample t tests with Bonferroni adjustment showed that participants scored statistically significantly higher on the measure of justification by authority (M = 7.03, SD = 1.61) than on the measure of justification by multiple sources (M = 6.22, SD = 1.70), t(63) = 3.00, p = 0.004, Cohen’s d = 0.49, and the measure of personal justification (M = 4.20, SD = 1.83), t(63) = 8.84, p = 0.000, Cohen’s d = 1.64. Moreover, participants’ scores on the measure of justification by multiple sources were statistically significantly higher than their scores on the measure of personal justification, with t(63) = 7.13, p = 0.000, Cohen’s d = 1.14. Consistent with our expectations, then, participants were most likely to believe in justification of knowledge claims in natural science by reliance on perceived experts and “scientificness” (Thomm and Bromme 2012), and they were least likely to believe in justification of knowledge claims by appeals to personal opinion.

Intercorrelations between the variables are shown in Table 3. To address our second research question, concerning the unique and relative contribution of the three types of justification beliefs to multiple-documents comprehension, we performed a forced-order hierarchical multiple regression analysis with scores on the multiple-documents comprehension measure as the dependent variable. In this analysis, scores on the topic knowledge measure were entered into the equation as a predictor in step 1. In step 2, scores on the measures of personal justification, justification by authority, and justification by multiple sources, respectively, were added to predict multiple-documents comprehension. According to Miles and Shevlin (2001), a sample size of 60 is sufficient to detect medium to large effects when only four predictors are used. As can be seen in Table 4, topic knowledge explained a statistically significant amount of variance in step 1, R 2 = 0.07, F(1, 61) = 4.43, p = 0.039. In the second step, the addition of the three justification belief measures resulted in a statistically significant 21 % increment in explained variance, with R 2 = 0.28, F change(3, 58) = 5.79, p = 0.002, after step 2. In this step, personal justification was a strong negative predictor of multiple-documents comprehension (β = −0.49, p = 0.000) and justification by multiple sources was a positive predictor (β = 0.30, p = 0.013). Consistent with our expectations, then, the more participants believed that justification of knowledge claims in natural science should be based on personal opinion, the poorer their multiple-documents comprehension; and the more they believed that justification should draw upon multiple sources of information, the better their comprehension performance. Please also note that we were able to explain a substantial portion of the variance in multiple-documents comprehension with the four predictors. According to the rule of thumb proposed by Cohen (1988), 28 % explained variance (Cohen’s f² = 0.39) is considered to be a large effect in multiple regression analysis.

Table 3 Zero-order correlations for measured variables
Full size table
Table 4 Results of hierarchical regression analysis for variables predicting multiple-documents comprehension
Full size table

Before performing the described multiple regression analysis, we examined the Read&Answer output to check that the participants had actually read the five documents, finding that all participants except two had accessed every segment of every document in the reading session. Of note is also that there were no statistically significant correlations between scores on any of the four predictor variables and the reading times for any of the five documents or for total reading time, with rs ranging from −0.16 to 0.10 (ps > 0.20).Footnote 3

Discussion

This study contributes uniquely to research on both epistemic beliefs and multiple-documents comprehension by informing about the relative importance that adolescents attach to different sources of justification in science, as well as indicating that particular beliefs about justification can improve the prediction of adolescents’ multiple-documents comprehension beyond that afforded by differences in prior knowledge about the topic of the documents.

Our first research question asked whether science-specific beliefs in personal justification, justification by authority, and justification by multiple sources differed in strengths among the participants. Our analyses comparing scores on the three justification belief measures showed that participants put most emphasis on justification by authority and least emphasis on personal justification, with justification by multiple sources falling in between.

That students put much more trust in sources that they consider to be authoritative, such as textbooks, teachers, and scientists, than in personal opinion, is consistent with other research within the domain of science (Bråten et al. 2011b; Muis et al. 2006). It is also consistent with the fact that, given the “division of cognitive labor” (Bromme et al. 2010) existing in the information society, people inevitably have to rely on external expertise rather than own opinion and experiences in many cases (Bromme et al. 2010; Chinn et al. 2011). Moreover, participants’ strong emphasis on justification by authority seems consistent with the finding that students justify knowledge claims in science by considering the “scientificness” of textual information, as signaled by scientific discourse conventions such as citations of scientists, descriptions of methods, and writing style (Thomm and Bromme 2012). With respect to the relatively low ratings for personal justification, on the other hand, students may well consider personal opinion and experiences as a relevant and valuable source when evaluating different ideas and perspectives encountered in literary works (e.g., novels and poems), but still consider it less relevant and valuable in the domain of science. This idea is consistent with other research concerning the domain-specificity of epistemic beliefs, where students have been found to rely more on personal experience and less on authority not only in the arts/humanities but also in the social sciences than in the natural sciences and mathematics (for review, see Muis et al. 2006). Moreover, a hitherto untested possibility is that students may come to rely more on justification by multiple sources in domains where learning to handle and interpret multiple source documents is pivotal, such as in history, law, and theology. Of late, researchers (Bråten et al. 2009b; Stahl and Bromme 2007; Trautwein and Lüdtke 2007) have demonstrated that students’ epistemic beliefs may vary even across topics within domains.

It is interesting to note that justification by multiple sources was less emphasized than justification by authority by the participants. This observation is not trivial, given that the national curriculum for secondary school put much emphasis on critical reading of multiple documents through the consideration of sources (Norwegian Ministry of Education and Research 2006). Moreover, the fact that project-based learning is widely used in Norwegian schools, often implying that students read and evaluate multiple information sources, does seemingly not hinder students to perceive authority as more important than multiple sources for evaluating knowledge claims in science. As suggested by one of the anonymous reviewers, however, it is also conceivable that asking students about justification for knowing in natural science within the context of their ordinary classrooms might have made them put more emphasis on justification by authority than they might have done in an out-of-school context. While it is not entirely clear why justification by authority should be more intertwined with learning in school that justification by multiple sources for our participants (see above), this possibility of context or situation specificity in epistemic belief assessment deserves further attention (Chinn and Buckland 2012).

Our second research question asked about the relative contribution of the three justification belief dimensions to multiple-documents comprehension. As expected, personal justification was a unique negative predictor of multiple-documents comprehension after variance associated with topic knowledge was removed, while justification by multiple sources was a unique positive predictor. Possibly, the reason why personal justification negatively predicted performance in this study, was that relying on personal opinion when evaluating differing knowledge claims located in multiple documents on a relatively unfamiliar topic may have led to superficial processing of document ideas. In accordance with several authors (Bråten et al. 2008; Coté et al. 1998; Goldman 2004), viewing the process of knowing as inherently subjective or personal may actually result in a mental representation characterized by lack of cohesion and integration because it is too loosely grounded in the meanings of the documents themselves.

On the other hand, the unique positive contribution of beliefs concerning justification for multiple sources is consistent with research on multiple-documents comprehension that highlights the importance of corroboration across multiple sources to building an integrated understanding (Bråten and Strømsø 2011; Stahl et al. 1996; Wineburg 1991; Wolfe and Goldman 2005). In fact, when Wineburg (1991) originally described the corroboration observed among expert historians working with multiple documents, he described an approach to multiple-documents reading that seems to presuppose beliefs in justification by multiple sources (Bråten et al. 2011a). In essence, corroboration involves comparing and contrasting information and establishing relations among descriptions and explanations contained in different documents (Wineburg 1991). Presumably, such corroboration of information across multiple documents may engage readers in bridging inferential processing that help them see patterns and construct a more complete and integrated understanding of the topic.

Of note is that even though participants reportedly relied most on authority when justifying knowledge claims in science, reliance on authority did not uniquely predict their comprehension performance. This finding seems consistent with prior research in both history and science, indicating that trusting the textbook when other primary and secondary sources are available is rather untypical of high-achieving individuals (Bråten et al. 2009b; Rouet et al. 1996; Wineburg 1991). Thus, a mismatch may exist between students’ high trust in a perceived authoritative source, such as the textbook, and the relatively meager return they get on this trust in terms of multiple-documents comprehension (Bråten et al. 2009b).

This does not necessarily mean, however, that some means of justification are universally more adequate than others. As posited within philosophically based multidimensional frameworks of epistemic cognition (Chinn et al. 2011; Greene et al. 2008), people may validly justify knowledge claims by drawing on different sources, including external authority and internal resources. Moreover, these frameworks suggest that individuals may coordinate different means of justification, suggesting that epistemic sophistication lies in a “flexible balancing” (Bråten et al. 2012) of different means of justification to fit particular purposes, materials, and contexts. In such a view, then, our findings only indicate that in the specific multiple-documents scenario that we created for this study, beliefs in personal justification were negatively and justification by multiple sources positively related to multiple-documents comprehension. To further test the generalizability of our findings, however, future work needs to include participants representing other student populations that read multiple conflicting documents on other topics for other purposes.

In addition to the aforementioned limitation with respect to the generalizability of our findings, it should obviously be cautioned against drawing causal conclusions based on our correlational data collected only 1 week apart. Thus, firmer causal statements regarding relationships between these dimensions of justification beliefs and multiple-documents comprehension must await further longitudinal or, preferably, experimental work. As it now stands, we also cannot exclude the possibility that variables that were not controlled for in the present study, for example related to cognitive ability or personality (e.g., need for cognition; Cacioppo and Petty 1982), may have caused scores on the belief measures and comprehension performance to relate as they did. Moreover, because prior beliefs about the controversial topic of the documents may have played a role in the reading task context that we created (cf., Kardash and Howell 2000; Murphy and Alexander 2004), such beliefs should also be controlled for in future research on this issue.

Although beyond the scope of the present study, the intriguing possibility that relations between epistemic beliefs and working with documents presenting conflicting information may be bidirectional rather than unidirectional also requires further work where participants’ epistemic beliefs are measured before and after the reading of conflicting materials and compared to those of controls reading consistent materials (e.g., see Kienhues et al. 2008, 2011).

Finally, our sole reliance on self-reports of epistemic beliefs may be considered a limitation because this methodology cannot really tell us how different epistemic beliefs or stances are enacted during reading to facilitate or constrain comprehension performance. Therefore, future research could profitably supplement or replace self-reported justification for knowing beliefs with think-aloud data on such beliefs in action during multiple-documents comprehension (cf., Bråten et al. 2012; Ferguson et al. 2012a). Accordingly, the justification beliefs that we measured in this study should be regarded as separate from processing because holding a particular belief, for example that knowledge claims need to be justified through evaluation and integration of multiple information sources, does not mean that a reader will necessarily use a corroboration strategy when working with multiple documents.

Despite the limitations, our findings may suggest some important implications for educational practice. First, they suggest that by encouraging students to draw on personal opinion and experiences when evaluating knowledge claims, a practice not uncommon in many classrooms, teachers may actually do students a disservice, at least when they work on multiple conflicting documents on a controversial science topic. More specifically, a classroom discourse pattern that allows students to take relativistic stances on complex and controversial science topics presented in multiple documents primarily on the basis of their personal opinions may just function to strengthen their pre-existing opinions (and sometimes, misconceptions) rather than promote learning and comprehension. Second, they suggest that in challenging science-reading contexts, students should rather be taught to consider which claims to believe on the basis of such processes as comparing, contrasting, and corroborating across multiple sources of information. In the classroom, it may thus be important that teachers try to scaffold a multiple-documents comprehension discourse pattern where it is not acceptable that students merely evaluate different ideas and perspectives by comparing each of them with their own opinion, but rather required that students compare what the different documents actually say as well as the evidence they offer for their claims. For example, when discussing in class that one document claims that sun exposure should be reduced because it can cause skin cancer whereas another document claims that sun exposure should be increased because it is an essential source of vitamin D, it is not students’ pre-existing opinions and experiences of sun exposure that should decide what to believe but rather a careful, individual and collective, examination of the claims and a weighing of the evidence to support them.