Introduction

Can alternative energy resources replace fossil fuels? Are cell phones safe for use? Should digital textbooks supplant print textbooks in schools? Laypeople who wonder about such current controversial topics often turn to the Internet for information. Yet, Internet sources are exceedingly diverse and vary in their underlying aims, viewpoints, reliability, and quality. Therefore, in order to understand, evaluate, and integrate online information, the sources of that information should be taken into account (Goldman and Scardamalia 2013; Brand-Gruwel and Stadtler 2011). Sourcing involves attending to and evaluating source information and using source information to interpret the content (Britt and Aglinskas 2002; Bråten et al. 2011a; Wineburg 1991). Prior research has repeatedly documented that children, youth, and adults infrequently engage in critical appraisal of sources and rely heavily on topic relevance, content adequacy, and design cues in order to evaluate information (e.g., Sundar 2007; Gasser et al. 2012; Flanagin and Metzger 2007; Britt and Aglinskas 2002; Eshet-Alkalai and Chajut 2009; Brand-Gruwel et al. 2009).

Sourcing can be especially challenging when learning about controversial issues in which there is open disagreement among experts. Laypeople often have limited conceptual knowledge on such complex issues, which reduces their ability to engage in direct “first-hand” evaluation of knowledge claims (Bromme and Goldman 2014). Consequently, their evaluation of competing accounts depends on “second-hand” evaluations of source reliability and quality (Bromme et al. 2010a; Bromme and Goldman 2014). However, evaluation of competing expert sources requires going beyond surface cues of expertise and authority and necessitates appraisal of additional indicators of expertise, such as experts’ professional backgrounds and track-records, viewpoints, interests, and biases (Goldman 2001). It is not clear how laypeople cope with this task and what is the extent of their ability to process and evaluate discrepant expert sources while reading.

Thus, a primary objective of the current study was to provide a detailed account of learners’ spontaneous sourcing practices as they engage in reading divergent expert accounts, in order to better understand how learners approach this task and to identify specific challenges they might face. Additionally, the study explored whether learners’ epistemic perspectives regarding knowledge and knowing are reflected in their approaches to sourcing and whether sourcing is related to written argumentation following reading. By attending to some of the precursors and corollaries of sourcing, we hoped to trace how spontaneous sourcing practices emerge and to evaluate their contribution to knowledge construction.

Sourcing and multiple document comprehension

Bråten and his colleagues defined multiple document comprehension as the “building of a coherent mental representation of an issue from the contents of multiple texts that deal with the same issue from different perspectives” (Bråten et al. 2013a). The need to represent and connect multiple documents in which diverse authors provide different accounts of the issue at hand makes sourcing vitally important for multiple document comprehension (Bråten et al. 2011a). In the present study, we employed the Documents Model framework (Rouet 2006; Britt and Rouet 2012) in order to conceptualize sourcing and to explicate its role in knowledge construction. To clarify the terminology used in this study, we use the terms “document” and “information source” interchangeably to refer to varied informational artifacts and reserve the term “source” for document features such as author backgrounds and viewpoints, document genre, and publication date (Britt and Rouet 2012).

In brief, the Documents Model framework extends the situation model theory of text comprehension (Kintsch 1988) to cases in which people read multiple documents that provide diverse accounts of the same situation. In such cases, readers need to construct a situations model, more recently termed an integrated mental model, that represents the agreements and discrepancies in the accounts they read (Britt et al. 2013; Britt and Rouet 2012). Furthermore, effective comprehension of multiple documents also entails construction of an additional layer of representation, called the intertext model, in which sources are represented and connected (Bråten et al. 2011a). Constructing the intertext model involves creating document nodes that represent the source of each document and contain information about the author, form, setting, and rhetorical goals of the document, as well as a summary of the key point or claim of the document (Britt and Rouet 2012). Additionally, the intertext model includes source-content links that connect between document nodes and document content (e.g., who says what), and source–source links that specify the relations between documents (e.g., support or opposition).

The Documents Model framework describes the representational structures that account for comprehension of multiple documents (Britt et al. 2013). Constructing a documents model involves several iterative processes that are detailed in the Multiple Documents—Task-based Relevance Assessment and Content Extraction (MD-TRACE) model (Rouet and Britt 2011). According to the MD-TRACE model, readers process sources in their initial assessment of document relevance in order to determine document reliability. Further processing of sources occurs as readers construct a documents model by creating links between sources and contents and forming connections across documents (for more detail see Rouet and Britt 2011). The processes involved in comprehending multiple texts require effort and motivational involvement because intertextual relations are often implicit and need to be constructed by readers (Bråten et al. 2014a).

Lay sourcing practices

Evidence suggests that learners are not always inclined or able to construct robust intertext models (Britt et al. 2013; Goldman 2004). Novices have generally been found to pay low attention to sources and to infrequently use source features to actively evaluate information sources (e.g., Britt and Aglinskas 2002; Wineburg 1991; Walraven et al. 2009; Goldman et al. 2012; Strømsø et al. 2013). However, lay readers can distinguish between various types of sources in their source evaluations and remember associations between source information and document contents (e.g., Rouet et al. 1996; Strømsø and Bråten 2014; Bråten et al. 2014b).

Students have been found to prefer information sources written by authors with high expertise over those written by authors with low expertise, particularly when students’ familiarity with the topic was low (McCrudden et al. 2015). However, authoritative texts, such as textbooks or expert essays, have been found to be evaluated less by author or publisher characteristics and more by content and document type characteristics, compared to less authoritative texts, such as participant accounts or newspapers articles (Rouet et al. 1996; Bråten et al. 2011b). This suggests that readers may have implicit trust in expert authors, which might lead them to prefer information sources written by experts and nonetheless to pay less attention to author characteristics when they process such information sources. Conflicts between information sources have been found to lead learners to pay more attention to document sources (Braasch et al. 2012; Strømsø et al. 2013) and thus might also help alert readers to differences between expert authors (Barzilai and Eshet-Alkalai 2015).

Explicit instructions to evaluate sources can also increase attention to source information and the frequency of source evaluations during reading in comparison to spontaneous (i.e., uninstructed) sourcing (Gerjets et al. 2011). However, because such instructions can make learners more aware of their evaluation processes they may distort sourcing processes (Gerjets et al. 2011). Not many studies have employed think-aloud techniques to examine concurrent spontaneous sourcing. In two such recent studies, Strømsø, Bråten and their colleagues investigated students’ spontaneous sourcing while reading conflicting documents of varying expertise and reliability regarding a socio-scientific controversy (Strømsø and Bråten 2014; Strømsø et al. 2013). The authors found that, on the whole, participants were more likely to simply pay attention to source information than to evaluate source or content credibility. Furthermore, source information was infrequently used to predict or interpret the content (Strømsø and Bråten 2014; Strømsø et al. 2013).

Epistemic perspectives and sourcing: Thinking about the sources of knowledge

Wineburg (1991) argued that sourcing is not simply a heuristic but rather a manifestation of beliefs regarding the nature of texts and the roles of authors. Indeed, subsequent studies have gone on to explore how views of knowledge and knowing might be related to sourcing (e.g., Whitmire 2004; Strømsø et al. 2011; Porsch and Bromme 2011). Epistemic thinking includes metacognitive knowledge, skills, and experiences regarding the nature of knowledge and knowing, as well as cognitive strategies and processes for reasoning about the epistemic characteristics of specific information, knowledge claims, and their sources (Barzilai and Zohar 2014, in press). Thus, sourcing can be viewed as a cognitive-level epistemic strategy because it involves reasoning about the epistemic properties of specific sources (e.g., their reliability). In contrast, beliefs and understandings about the nature of knowledge and knowing, including issues such as the certainty, sources, structure, and justification of knowledge, are a meta-level epistemic knowledge (Kuhn 2001). This epistemic metacognitive knowledge has domain-general as well as domain- or topic-specific aspects and may inform and guide the performance of epistemic strategies, such as sourcing (Barzilai and Zohar 2014, in press).

Bråten et al. (2011) pointed out that epistemic beliefs about the nature knowledge and knowing can shape multiple document comprehension in several ways: Epistemic beliefs may be related to readers’ perceptions of the task and its goals, to the standards and criteria they adopt for task performance and completion, and to the strategic processes they employ to meet task goals and standards. Indeed, several studies have documented that beliefs that knowledge is complex, evolving, and justified through inquiry and corroboration of multiple sources are related to better multiple-document comprehension (reviewed in Bråten et al. 2011a; Ferguson 2014). Yet, epistemic beliefs in the knower as a constructor of knowledge and in personal justification have been found to be negative predictors of multiple-document comprehension (Strømsø et al. 2008; Bråten et al. 2008, 2013b).

More specifically, epistemic beliefs have been found to be related to source use and evaluation (Whitmire 2004; Bråten et al. 2014b; Kammerer et al. 2013; Strømsø et al. 2011; Porsch and Bromme 2011). For example, readers who believe that the source of knowledge is in personal judgments and opinions were found to trust documents less and to rely more on their own opinion as a trustworthiness evaluation criterion (Strømsø et al. 2011). In contrast, beliefs that knowledge should be justified by critical evaluation and corroboration predicted greater trust in a reliable text and more reliance on author and content evaluation criteria (Strømsø et al. 2011). Additionally, learners’ beliefs in the Internet as a reliable resource of accurate and factual knowledge were found to predict less attention to source information while reading online and less verbal reflections on source credibility (Kammerer et al. 2013).

In the present study, we did not conceptualize epistemic thinking as a system of beliefs but rather approached students’ epistemologies as integrated positions, employing the model proposed by Kuhn and her colleagues (Kuhn and Weinstock 2002; Kuhn 2001, 1991; Weinstock 2009). This model argues that epistemic thinking is a developing “theory-in-action” that emerges in multidimensional forms when people reason about specific knowledge claims and information sources (Kuhn and Weinstock 2002). Thus, according to this view, epistemic thinking is not a static approach but rather varies across tasks and domains (Kuhn et al. 2000, 2008).

The Kuhn et al. model describes three main epistemic positions or perspectives: An absolutist perspective that knowledge is objective, located in the external world, and certain; a multiplist perspective that the source of knowledge is in individuals and that knowledge is therefore subjective and uncertain; and an evaluativist perspective that considers knowledge as constructed and acknowledges uncertainty without forsaking the need for evaluating knowledge production. These epistemic perspectives entail different views of expertise (Kuhn 1991; Kuhn and Weinstock 2002): Absolutism is associated with an assumption that reliable and trustworthy experts can know with certainty; Multiplism is associated with a denial of the possibility of expert certainty; Finally, evaluativism is associated with the view that although certainty is difficult to attain, experts can have greater certainty than the average person. These perceptions of the sources, justification, and limits of knowing could be expected to inform learners’ sourcing practices.

In order to examine this assumption, Barzilai and Zohar (2012) compared how sixth-graders, who expressed absolutist and evaluativist views regarding a particular topic, evaluated the trustworthiness of online information sources on that topic. Somewhat unexpectedly, they found that absolutist and evaluativist perspectives were unrelated to the level of engagement in trustworthiness evaluation. Explaining this finding, Barzilai and Zohar (2012) proposed that students who endorse absolutists and evaluativists views are similarly likely to think critically about sources but may grasp the aims of source evaluation differently: From an absolutist approach, the aim of evaluation might be deciding whether a source is reliable or not, based on source credentials, expertise, potential bias, etc. In contrast, from an evaluativist approach, the aim of evaluation might be to weigh different viewpoints regarding the issue at hand by considering source reliability as well as source backgrounds and positions. Indeed, in a subsequent study, Barzilai and Eshet-Alkalai (2015) demonstrated that absolutism and multiplism are negative predictors, and that evaluativism is a positive predictor, of comprehension of multiple author viewpoints regarding a socio-scientific controversy. These findings suggest that epistemic perspectives may subtly impact the ways in which sources are considered and evaluated.

Sourcing and argumentation: Coordinating multiple accounts

Detailed intertext models, in which sources are represented and connected, are posited to contribute to knowledge construction by facilitating a more coherent representation of multiple documents (Britt and Rouet 2012; Bråten et al. 2009). In this study, we were specifically interested in examining the relation between sourcing while reading and the quality of learners’ arguments following reading. An argument minimally includes a claim supported by at least one relevant reason and an often implicit warrant that supports the relation between reason and claim (Means and Voss 1996; Toulmin 1958/2003). However, well-reasoned arguments regarding complex problems also include consideration of possible counter-arguments (Means and Voss 1996; Kuhn 1991). Learners’ ability to produce arguments that integrate multiple positions or accounts can indicate the depth of their reasoning regarding the issue and the extent to which they evaluate and weigh alternatives in order to justify a resolution (Nussbaum and Schraw 2007).

How might sourcing contribute to argumentation? Britt and Rouet (2012) proposed that as learners read multiple documents they need to represent and relate competing theories or accounts and the evidence that supports them. This process ideally revolves around an argument schema that organizes information from documents by identifying claims, supporting reasons, oppositions, and limitations (Britt and Rouet 2012; Bråten et al. 2011a). Presumably, the more learners are attentive to sources and their connections, the better they may understand the similarities, differences, and relations among multiple accounts of a problem, leading to a more coherent understanding of the problem and better integration of multiple accounts in learners’ arguments (Bråten et al. 2014a; Anmarkrud et al. 2014). Furthermore, learners’ specific source evaluations, e.g., evaluations of source credibility, can also impact the ways in which learners resolve conflicts between accounts (Kobayashi 2014; Bråten et al. 2014a).

Not many studies have directly examined the contribution of sourcing while reading multiple documents to subsequent argumentation. In one recent study, source and content evaluations and inter-textual linking processes while reading were found to be related to more elaborate argument structures, increased source citations, and more source-content links in participants’ argument essays (Anmarkrud et al. 2014). In a similar vein, comprehension of author viewpoints was found to predict increasingly complex arguments that integrated more information sources (Barzilai and Eshet-Alkalai 2015). Source references and source-content links in students’ essays were also found to be positively correlated with argument complexity (Bråten et al. 2014b). The relation between sourcing and claim justification is less clear. In one study, higher frequencies of source evaluations while reading were not found to be related to the quality of justifications for a subsequent decision (Gerjets et al. 2011). Based on these conflicting results, there is still a need to further examine if and how sourcing contributes to various dimensions of argument construction.

The present study

Despite the crucial role that sourcing plays in comprehension of diverse information sources there is still insufficient evidence regarding whether and how learners spontaneously construct intertext models as they read. Furthermore, sourcing studies typically present readers with information sources of varying levels of authority and expertise (e.g., Goldman et al. 2012; Strømsø et al. 2013). However, in controversial contexts learners may encounter conflicting expert accounts and need to employ strategies to deal with disagreements among expert sources (Thomm et al. 2015). Therefore, our first research question was: How do learners spontaneously attend to source information and process that information while reading divergent expert information sources about a socio-scientific controversy? Specifically, how do learners form source representations, source-content links, and source–source links while reading?

Epistemic thinking has been proposed to play an important role in shaping learners’ sourcing practices (Bråten et al. 2011a; Wineburg 1991). Yet, as Bråten et al. (2014) have noted, the paths between epistemic thinking and comprehension of multiple documents are not well charted. Specifically, a previous think-aloud study suggested that absolutist and evaluativist epistemic perspectives might not necessarily be reflected in the extent of sourcing but rather in the ways in which learners in engage in sourcing (Barzilai and Zohar 2012). Furthermore, there is scant documentation of the role of multiplism in sourcing. Therefore, a second aim of the study was to better clarify the role of epistemic perspectives in sourcing. Specifically, the second research question was: Are learners’ absolutist, multiplist, and evaluativist epistemic perspectives related to their spontaneous sourcing while reading divergent expert information sources, and if so, how?

Previous studies have found that better sourcing is sometimes related to more complex argumentation (Anmarkrud et al. 2014; Barzilai and Eshet-Alkalai 2015; Bråten et al. 2014b). In this study, we sought to extend current understandings of the connection between sourcing and argumentation by examining more closely how specific sourcing activities might be related to written argumentation. Thus, our third research question was: Is learners’ sourcing while reading divergent expert information sources related to their subsequent arguments, and if so, how?

Topic knowledge and topic interest have been found to be positively related to multiple document comprehension (e.g., Strømsø et al. 2010; Rouet et al. 1997). Therefore these variables were assessed in order to control for their potential effects. We also examined gender as a possible covariate because in two prior studies men were found to score higher than women on intertextual comprehension tasks (Strømsø et al. 2008; Bråten and Strømsø 2010).

Method

Participants

Sixty-one Hebrew-speaking university students (39 women and 22 men) participated in the study. Their mean age was 30.50 years (SD = 8.44). The age range of the participants was relatively large because the study was conducted among students of an open university that offers distance learning courses. Participants were BA students (84 %) or MA students (16 %). Most participants (80.3 %) were studying toward degrees in social sciences (mainly psychology and education). The remaining participants were studying toward degrees in humanities and arts, management and economics, or computer science. Almost all participants (95.1 %) had an Internet connection at home. Some students participated in the study as part of their degree requirements, while others were recruited through ads and were compensated for their participation.

Materials

The topic chosen for this study was the economic and environmental implications of seawater desalination. In recent years, the use of seawater desalination technologies for water production in Israel has grown rapidly, and desalination plants currently supply more than 50 % of Israel’s potable water. Nonetheless, there is very little public debate about desalination in Israeli media, and public awareness of the possible implications of desalination is low. Consequently, we did not expect participants to have a high level of prior knowledge or strongly held convictions about the topic.

Blogs have become an increasingly common form of science communication (Brumfiel 2009). Therefore, we used four designed blog-posts in order to introduce the controversy. The blog-posts were written based on a comprehensive survey of online information sources about seawater desalination, and presented arguments that reflected the positions of actual experts and stakeholders. The authors of the blog-posts were presented as PhD-level experts who work as consultants to various government agencies that are stakeholders in the national water policy. The experts presented conflicting economic and environmental arguments for and against desalination. One economist and one hydrologist raised arguments in support of desalination and one economist and one hydrologist raised arguments against desalination. Content validity of the blog-posts was examined by two content experts.

Each text included, from top to bottom: the blog-post title; the author’s name (e.g., Dr. Rabinovitz), profession (e.g., hydrologist), and affiliation (e.g., consultant to the National Water Authority); and the body of the text. Each text presented one main argument concerning desalination, supported by explanations and data. The blog-posts were of similar length (M = 213.7 words, SD = 9.7) and writing style. The blog-posts were written in clear language and avoided technical jargon. An overview of the blog-posts is provided in Table 1. The texts of blog-posts 1 and 2 are provided in an online supplement. Additional materials are available from the authors upon request.

Table 1 Blog-post descriptions
Full size table

Measures

Epistemic thinking assessment

Epistemic thinking was assessed using a scenario-based epistemic thinking assessment that referred to the desalination issue (Barzilai and Weinstock 2015). The assessment is a topic-specific measure that assesses absolutist, multiplist, and evaluativist perspectives by probing participants’ epistemic metacognitive knowledge regarding the attainability of truth and the nature, sources, certainty, justification, and reliability of knowledge. Participants first read an introduction that described the current state of seawater desalination in Israel, and stated that scientists are investigating the environmental impacts of seawater desalination. Participants then responded to a series of 12 epistemic questions that referred to the topic of desalination. Each question was followed by three items that reflect typical absolutist, multiplist, and evaluativist responses. Participants were asked to rate their agreement with each item on a ten-point scale (from very much disagree to very much agree). Table 2 includes examples of three questions and their respective items.

Table 2 Sample questions and items in the scenario-based epistemic thinking assessment (Barzilai and Weinstock 2015)
Full size table

Due to sample size limitations, the epistemic perspective scales were constructed based on a principal component analysis conducted by Barzilai and Eshet-Alkalai (2015). Following the results of this analysis, only 28 items, which exhibited adequate loadings on their respective factors, were used in the current study. Internal consistency of the scales in the current sample was satisfactory: Absolutism, 10 items, Cronbach’s α = .87; Multiplism, 8 items, Cronbach’s α = .85; and Evaluativism, 10 items, Cronbach’s α = .83. Scores for each scale were calculated based on the mean of the items.

Three epistemic perspectives scores were retained per participant. Although participants may endorse a predominant epistemic perspective, they might also endorse other epistemic perspectives to some extent. This could indicate transition between perspectives or might reflect ongoing conflicts in epistemic thinking (e.g., Feucht 2011). In particular, rather than a static and settled position, evaluativism might better be characterized as an ongoing attempt to coordinate the objective and subjective dimensions of knowing. The retention of three epistemic perspective scores per participant can thus provide a fuller account of participants’ epistemic views.

Topic knowledge measure

Topic knowledge was assessed using a multiple-choice test composed of 12 statements that related to various claims made in the texts. A sample item is: “In the past decade, water consumption per capita in Israel has declined” [incorrect]. Participants judged whether each statement was correct or incorrect, and they could also indicate that they do not know the answer. The items of the measure reflected diverse areas of knowledge that were discussed in the blog-posts, such as water economy and ecology. Therefore, test–retest reliability was a more appropriate reliability indicator, for this measure, than internal consistency reliability. Test–retest reliability was examined in an independent sample, N = 77, with two weeks between test and retest, and was found to be r = .73. The topic knowledge score was calculated using the sum of the correct responses.

Topic interest measure

Participants’ interest in the topic was assessed using a 10-item questionnaire developed by Mason, Gava, and Boldrin (2008), which had been translated to Hebrew and adapted to the topic of desalination by the authors. A sample item is: “I am keen to know about water desalination.” Items were scored on a six-point scale (from very much disagree to very much agree). Internal consistency reliability in the current sample was Cronbach’s α = .92. The topic interest score was based on the mean of the items.

Argument task

After reading the blog-posts, participants were asked to write an argument concerning desalination. The writing instructions were: “Please write an argument that addresses the question: Should the State of Israel continue to encourage the construction of seawater desalination plants? Present and justify your position on this issue.” Participants did not have access to the blog-posts while writing their arguments. The analysis of the arguments took into account several dimensions that were identified in prior studies as indicative of good argumentation (Schwarz et al. 2003; Means and Voss 1996; Zohar and Nemet 2002) and of multiple information source integration (Gil et al. 2010; Goldman et al. 2013). Argument structure was scored by awarding one point for each of the following elements: a claim supported by at least one relevant reason, a qualification of the claim, a counter-claim, and counter-claim justification. Additionally, we counted the number of relevant reasons and awarded one point per reason provided (up to four points), since a greater number of acceptable reasons indicates a stronger argument (Means and Voss 1996). Finally, we awarded one point per each information source that was reflected in the argument in order to assess how well the argument integrates information from multiple accounts (Gil et al. 2010; Goldman et al. 2013). Because participants did not typically make explicit references to the blog-posts in their arguments, we created a list of unique ideas for each blog-post and identified these ideas in the arguments. The argument coding scheme, including examples and scoring, is detailed in Table 3. The arguments were coded by the first two authors. Inter-rater reliability was tested using 40 arguments. Cohen’s kappa of the argument structure, argument reasons, and argument sources codes was .92, .81, and .74, respectively.

Table 3 Argument coding scheme
Full size table

Procedure

The think-aloud method (Ericsson and Simon 1993) was used to document sourcing while reading. The think-aloud instructions did not include explicit sourcing prompts, but rather asked the participants to say everything they think and do while reading. The choice of this type of instructions was guided by the observation that explicit sourcing instructions can increase sourcing levels and impact the ways in which participants source (Gerjets et al. 2011). By not directing the participants to engage in sourcing we hoped to provide a more accurate documentation of spontaneous sourcing. The think-aloud instructions are provided in full in an online supplement.

Data were collected by the second author in a university computer lab. All of the measures and blog-posts were displayed on a computer using survey software. Participants first completed a demographic questionnaire, a computer and Internet use survey, and the epistemic thinking, topic knowledge, and topic interest measures. They were then trained to think aloud using two informational texts about unrelated topics (the health effects of high pressure and coffee drinking). After training, participants were told that they would read four blog-posts about seawater desalination in Israel that contain information that could help them form a position on this topic, and that they would subsequently be asked to present and justify their position on desalination in writing. Participants then read the blog-posts while thinking aloud. The blog-posts were presented consecutively on separate pages and in random order. After reading, participants wrote arguments concerning seawater desalination. All sessions were audio-recorded and fully transcribed.

Sourcing coding scheme

The sourcing coding scheme was based on a scheme by Strømsø and colleagues, which was expanded in light of the current data (Strømsø et al. 2013; Strømsø and Bråten 2014). The unit of analysis was defined as a comment or a set of comments that relate to specific source information (Strømsø et al. 2013). As in Strømsø et al.’s scheme, we identified three types of sources mentioned in participants’ comments: the present blog-post, other blog-posts in the set, or other sources not included in the document set (The documents did not include embedded sources). We identified three types of sourcing activities that involved these source types:

  • Source representations characterized the ways in which sources were described by readers. We noted if participants made explicit references to sources by stating the source name or characteristics, or if they noted the sources only implicitly without precisely naming or characterizing them (Strømsø et al. 2013). Additionally, we coded if participants were simply paying attention to source information or if they were also explicitly using source information to evaluate the source (Strømsø et al. 2013). Finally, we also coded the specific source characteristics that participants mentioned.

  • Source-content links mapped the various types of connections made between sources and the content of the blog-posts. Following Strømsø et al. (2013), we coded all instances in which participants used source information to predict, interpret, or evaluate the content. We also added a new “connecting” code that captured all of the simple connections made between sources and contents (i.e., who-said-what).

  • Source-source links described the ways in which readers compared and contrasted sources. This category was newly added in the current study and included comparisons of source claims, source positions or perspectives, other source properties, and source reliability.

Definitions of the codes along with examples are provided in Table 4. Each sourcing comment was coded using multiple codes, when applicable.

Table 4 Sourcing coding scheme
Full size table

In the analysis of the protocols, we noticed that participants frequently reflected on the availability, sources, and adequacy of their own knowledge about the topic. In light of previous findings regarding the negative relations between beliefs in personal justification and multiple document comprehension (e.g., Bråten et al. 2013b) and the positive relation between beliefs in personal justification and reliance on one’s own opinion for evaluating texts (Strømsø et al. 2011), we decided to code these utterances. The code “the reader as a source of knowledge” was defined as an explicit reflective reference to the reader’s knowledge of the topic, e.g., “I remember that after the last winter, the water level in the Sea of Galilee actually rose” [P41]. Such coded utterances sometimes included elaborations of the content based on prior knowledge, yet such elaborations were regarded as references to the reader as a source of knowledge only when they included metacognitive monitoring of the reader’s prior knowledge on the topic.

The first two authors independently coded 30 protocols in order to test the interrater reliability of the coding scheme. Reliability of the codes ranged from Cohen’s kappa .87 to 1.00.

Analyses

Most of the sourcing variables exhibited substantial departure from normality. Hence, we employed Wilcoxon signed ranks tests to compare dependent-sample means and Mann–Whitney tests to compare independent-sample means, reporting the r statistic as a measure of effect size (Field 2013). Correlations were tested using Spearman’s correlation coefficient. The epistemic perspectives and argumentation variables were approximately normally distributed and were therefore also analyzed using parametric tests. This quantitative analysis is complemented by a narrative analysis of two contrasting cases of participants who exhibited different approaches to sourcing.

Results

Overview of the sourcing comments

On average, participants made 3.59 sourcing comments, SD = 4.38 (0.90 comments per blog-post). Sourcing comments sometimes referred to more than one information source. The source of the present blog was mentioned most frequently, M = 3.46, SD = 4.36. Sources of other blog-posts were mentioned much less often, M = .89, SD = 1.28. Still fewer references were made to other sources not included in the document set, M = .13, SD = .34. Table 5 includes the frequencies, percentages, and means of all sourcing utterances.

Table 5 Descriptive statistics of the sourcing utterances
Full size table

Source representations

By definition, all sourcing comments included some representation of sources. Participants mostly referred to sources in implicit ways, M = 2.85, SD = 3.67, and were less likely to make explicit source references, M = 0.74, SD = 1.34, z = 5.00, p < .001, r = .64. Participants usually simply paid attention to the source, M = 3.05, SD = 3.62. Evaluations of source reliability were comparatively infrequent, M = .54, SD = 1.53, z = 5.29, p < .001, r = .67. The source characteristics commented on most frequently were source position, perspective, and expertise. Participants less frequently referred to currency and motivation, and seldom mentioned other source characteristics. Figure 1 presents the percentage of participants who mentioned each source characteristic at least once while reading.

Fig. 1
figure 1

Spontaneous references to source characteristics (percentage of participants who mentioned each characteristic at least once)

Full size image

Source-content links

Source representations were usually connected to document content and 88.6 % of the sourcing comments included such links. The predominant source-content link activity was creating simple connections between sources and claims, i.e., noting who says what, M = 2.72, SD = 3.09. Use of source information to evaluate content reliability was a much less frequent activity, M = .44, SD = 1.18, z = 5.64, p < .001, r = .72. With a single exception, participants did not use source information to explicitly predict or interpret blog content.

Source-source links

Participants were less likely to describe connections between sources and only 22.8 % of the sourcing comments included source–source links. The most frequent source–source activity was comparing source claims, M = 0.46, SD = 0.91. Participants less frequently compared source positions or perspectives, M = 0.25, SD = 0.57, z = 2.07, p = .038, r = .27, or other source characteristics, M = .10, SD = .30, z = 2.07, p = .039, r = .26. Only one participant explicitly compared source reliability.

The reader as a source of knowledge

References to the reader as a source of knowledge were made by 63.9 % of the participants, M = 1.38, SD = 1.44 comments per participant. References to the reader as a source often included reflections on the origins of the readers’ knowledge and its adequacy for evaluating the content. Readers’ knowledge was also frequently used to question, reject, or confirm knowledge claims made in the documents, e.g., “This is something I know. Prices really have gone up” [P6].

Sourcing profiles

The low average levels of spontaneous sourcing can mask considerable variability among participants. To better understand this variability, we examined how many participants engaged in constructing source representations, source-content links, and source–source links. This theoretically-guided analysis resulted in four sourcing profiles that could be identified in the data:

  • Profile A: No sourcing 12 participants (19.7 %) made no sourcing comments at all.

  • Profile B: Minimal sourcing 2 participants (3.3 %) mentioned source representations only and made no explicit source-content or source–source links.

  • Profile C: Low sourcing 20 participants (32.8 %) mentioned source representation and source-content links but did not describe any source–source links. For example, the following participant referred to the source of each blog but did not compare sources:

    “[Reads blog-post 2] It is not clear to me if when she speaks of alternative sources she is speaking about desalination or something else… OK, so this is actually pro desalination… [Goes on to read blog-post 3] I agree with what is written here… I’m not sure this is sufficient, such a solution…. OK… What he says is interesting…” [P40].

  • Profile D: High sourcing 27 participants (44.2 %) mentioned source representations, source-content links, and source–source links. For example, the following participant reflected on the sources’ characteristics and claims while reading and also actively compared source claims across documents:

    “[Reads blog-post 2] It seems as if she has a vested interest here because she is a consultant to the Water Authority… Compared to the previous one, they didn’t mention the [pollution of the] water wells in the Judea and Coastal Plain area. … It is clear that she supports desalination… [Goes on to read blog-post 3] … This doesn’t make sense, in the first blog they mentioned different data and the numbers don’t work out… Here they don’t say anything about desalination and they do talk about other solutions…” [P7].

Thus, substantial differences in intertext model construction were found. Some participants did not visibly engage in constructing intertext models, some participants expressed partial intertext models, and other participants spontaneously described the full range of representations that are entailed in forming an intertext model. Figure 2 compares the mean number of sourcing activities in each sourcing profile. High sourcing participants were also found to mention considerably more source representations and source-content links than low sourcing participants, z = −3.62, p < .001, r = 0.53, and z = −3.16, p = .002, r = 0.46, respectively. Notably, participants who did not engage in sourcing made more comments referring to the reader as source of knowledge, M = 2.25, SD = 1.82, than participants who engaged in at least some sourcing, M = 1.16, SD = 1.26, z = 1.99, p = .047, r = .25.

Fig. 2
figure 2

Mean number of sourcing activities per sourcing profile

Full size image

Relations between epistemic perspectives and sourcing

A multinomial logistic regression indicated that epistemic perspectives were not significant predictors of sourcing profile. Further examination of the correlations between epistemic perspectives and sourcing, among participants who engaged in sourcing, revealed that absolutism was marginally positively correlated to the number of source representations, r s  = .25, p = .089. Multiplism was negatively correlated to source representations, r s  = −.31, p = .031, and source-content links, r s  = −.31, p = .028, and positively related to references to the reader as a source, r s  = .36, p = .012. Evaluativism was also positively related to references to the reader as a source, r s  = .29, p = .040, but was not significantly related to any other sourcing variable. Descriptive statistics of the epistemic perspective variables are provided in Table 6, and all correlation coefficients are presented in Table 7. Additionally, we found that, among sourcing participants, multiplism was negatively correlated to the total number of references to source positions and perspectives, r s  = −.31, p = .032, and to source currency, r s  = −.33, p = .020.

Table 6 Descriptive statistics of the topic knowledge, topic interest, argumentation, and epistemic perspective variables among all participants (N = 61)
Full size table
Table 7 Spearman correlations among epistemic perspectives, sourcing, and argumentation among participants who engaged in sourcing (n = 49)
Full size table

Additional predictors of sourcing

Participants’ topic knowledge was rather low and was not significantly correlated to sourcing activities. Topic interest was moderate and was negatively correlated to references to the reader as a source of knowledge, r s  = −.29, p = .022. Unexpectedly, gender emerged as a predictor of sourcing profile, χ2 (2, N = 59) = 6.80, p = .033, Cramer’s V = .34: 29.7 % of the women and only a single man were included in the no sourcing profile, 35.1 % of the women and 31.8 % of the men were included in the low sourcing profile, and 35.1 % of the women and 63.6 % of the men were included in the high sourcing profile. No significant gender differences were found in age, educational level, reported frequency of Internet use, topic knowledge, topic interest, or epistemic perspectives.

Relations between sourcing and written argumentation

ANOVAs with sourcing profile (high, low, and no sourcing; the minimal sourcing profile was excluded because it included only two participants) as the independent variable and argument structure, argument reasons, and argument sources as dependent variables, revealed no significant effects of sourcing profile on argument reasons and argument sources and a significant effect of sourcing profile on argument structure, F(2,56) = 5.69, p = .006, η 2p  = .17. Post-hoc Tukey HSD tests indicated that high sourcing participants produced significantly more complex arguments, M = 2.33, SD = 0.96, than low sourcing participants, M = 1.45, SD = 1.05, p = .010, and no sourcing participants, M = 1.50, SD = 0.91, p = .046. No differences were found between no sourcing and low sourcing participants. Descriptive statistics of the argumentation variables are included in Table 6.

To explore which specific sourcing activities might contribute to argumentation, we examined the correlations between sourcing and argumentation among participants who engaged in sourcing. Source representations and source-content links were found to be positively correlated to argument structure, argument sources, and the number of reasons supporting the argument. Source-source links were positively correlated to argument structure only. References to the reader as source of knowledge were negatively correlated to argument reasons and argument sources. Correlation coefficients are exhibited in Table 7.

Interplay of epistemic perspectives, sourcing, and argumentation: two contrasting cases

To further shed light on our findings, we next analyze two contrasting cases from both ends of the sourcing spectrum: The first case is a participant who did not engage in sourcing and the second case is a participant who was classified in the high sourcing profile. These cases were chosen because they illustrate how learners’ epistemic perspectives may be related to their approaches to sourcing and how sourcing may shape subsequent argumentation.

Case 1: “As far as I know”

Anna (P23, pseudonym) was a 27 year old female who was studying toward a bachelor degree in humanities. She had a computer at home and reported using the Internet frequently for email, social networking, and information seeking. Her interest in the topic was moderate, 3.10 out of 6.00, and her prior knowledge was rather low, 5.00 out of 12.00, similarly to most of the participants in the study. Anna’s multiplism score was 10.00 out of 10.00 and her absolutism and evaluativism scores were 1.60 and 5.80 out of 10.00, respectively. Thus, Anna expressed very strong beliefs that knowledge on the topic of desalination is uncertain and that justification should be based on personal knowledge and opinions.

Anna’s commentary, as she read the documents, indicated that her primary strategy while reading was comparing her prior knowledge and experiences with the claims stated in the blog-posts. Anna frequently reflected on herself as a source as knowledge. Her very first comment while reading was, “I studied about this topic.” Anna also drew on her personal experiences as a source of knowledge, e.g., “As far as I know, in the last year or two the winters were really good. There was lots of rain. That should have made [water prices] lower, but right now they are only rising.” Anna was very attentive to inconsistencies and consistencies between her own knowledge and what she was reading, e.g., “This pretty much contradicts what I thought,” “Yes, I remember this from geography class.” Thus she was primarily engaged in “first-hand evaluation” (Bromme et al. 2010a) of the claims she was reading based on her prior knowledge. Throughout the protocol, Anna did not make a single reference to the sources of the blog-posts and did not consider the consistency of their claims.

Anna’s argument echoed the prior knowledge and personal concerns she had expressed while reading: “I think that the State of Israel should continue to encourage the construction of desalination plants because it is supposed to reduce our expenses. However, we shouldn’t rely only on that, but rather raise public awareness of the importance of saving water.” The argument included a claim supported by a single reason and a counter-claim that was not supported by reasons, and was therefore not a very elaborate or well-justified argument. The ideas Anna integrated in her argument were ones that were already well-connected to her prior knowledge and experiences. Thus, her argument only reflected claims that appeared in the two blog-posts that were written from a primarily economic perspective and did not refer to the environmental considerations raised in the two other blog-posts.

In summary, Anna’s case demonstrates how beliefs in the subjective nature of knowledge may be related to reliance on personal sources of knowledge as the pivot of knowledge evaluation and construction and hence to low attention to the voices and roles of external sources of knowledge. An emphasis on personal justification, coupled with low awareness of the diversity of positions regarding the topic, may result in arguments with minimal justification and low integration of multiple accounts.

Case 2: “I don’t know if he is right or not”

Dor (P28, pseudonym) was a 24 year old male who was studying for a bachelor degree in psychology, sociology, and anthropology. His reported Internet uses were very similar to Anna’s. His prior knowledge was slightly lower than Anna’s, 4.00, and his topic interest was somewhat higher, 4.40. Dor endorsed absolutism and evaluativism to a similar extent, 7.70 and 7.50 respectively, and endorsed multiplism to a lesser degree, 5.38. Dor’s high endorsement of evaluativism indicates that he was aware that the problem has both objective and subjective dimensions, but his high absolutism score suggests that he was also reluctant to relinquish the ideal that knowledge about the problem should be factual and certain. Hence, Dor’s epistemic position appeared to reflect an unresolved tension between his beliefs in the objectivity of knowledge and an emerging awareness that such objectivity might be unattainable.

While reading the blog-posts, Dor was attentive to the voices of sources. His first comment while reading immediately linked between source and content, “I think he really gives an introduction to what is going on in the country,” indicating that Dor noticed the source early in the reading process. He made similar implicit source-content links while reading all four blog-posts, thus expressing awareness of the roles of texts as constructed artefacts (Britt et al. 2013). Although Anna and Dor had roughly similar prior topic knowledge, Dor seemed more aware of the limitations of his knowledge, “I don’t really have information about desalination,” and did not rely as much as Anna on his personal sources of knowledge. Dor was concerned with source reliability, or, more precisely, with his difficulties in ascertaining source reliability, e.g., “I don’t know if he is right or not.” He also reflected on the considerations that underlie his source credibility judgments, “When I see people who are consultants I tend to trust them more than people who are in public service.” Importantly, Dor compared and contrasted sources while reading, e.g., “The first researcher said that [desalination] is expensive and that [water] wells are better and here I read… OK, this shows desalination in a positive light.” Hence his primary strategies for dealing with divergent accounts were based on “second-hand evaluation” (Bromme et al. 2010a) of the reliability of sources combined with “first-hand evaluation” of the consistency and plausibility of their arguments.

After reading, Dor wrote a complex argument that considered and justified both sides of the controversy and also attempted to reconcile them: “On the one hand, desalination has a positive aspect, and on the other hand, negative. My opinion is that we need to find the best solution, and when I say best I mean a solution that will cause minimal damage to nature while balancing water sources. … If we will desalinate water we will harm the ocean and if we will not desalinate we will dry up important water reservoirs….” The environmental and economic considerations raised in the argument reflected ideas from three of the blog-posts. In his argument, Dor also explicitly described how comparisons among sources evoked epistemic doubt and an awareness of the diversity of knowledge: “My opinion is conflicted. … I found it hard to base my opinion on what was written because many positions and opinions were presented.”

In summary, Dor’s reading of the texts reflected the tensions of his epistemic perspective: a commitment to seeking objectively reliable sources and arguments along with a disconcerting awareness of the difficulties of establishing reliability due to the subjectivity and multiplicity of sources. His case suggests two complementary paths through which sourcing might impact subsequent argumentation. First, attention to differences among authors’ backgrounds and viewpoints may help develop an understanding of the complexity of the issue at hand and an appreciation of the need to take multiple accounts into consideration. Second, attention to conflicting voices can evoke epistemic doubt that might lead to engagement in resolution strategies (Bendixen and Rule 2004) such as weighing and reconciling divergent accounts.

Discussion

Spontaneous sourcing practices: Widening the lens

Lay sourcing is often described as troublingly low (e.g., Brand-Gruwel and Stadtler 2011). Indeed, an analysis of sourcing activity means is likely to leave that impression. An important finding of this study is that spontaneous sourcing spans a wide range of sourcing practices: Some participants in our study did not visibly engage in sourcing, whereas others actively and systematically constructed intertext models while reading. Thus, the results of this study indicate that learners can construct complex intertext models, although only a minority appears to do so. Documenting the variability of learners’ sourcing practices may offer an informative approach for better understanding these practices, identifying the resources that learners bring to this complex task, and tracing possible causes of variability.

Despite this encouraging observation, our findings also point to a persistent gap between identifying and noting source information, and elaborating on that information and using it to evaluate and interpret sources and content. In contrast to previous sourcing studies (Strømsø and Bråten 2014; Strømsø et al. 2013), the participants in our study made more implicit than explicit sourcing comments. This may have been partially due to the absence of embedded sources within the documents (cf. Strømsø et al. 2013); The inclusion of sources in the body of the text might increase the likelihood of their being explicitly addressed as participants process the contents of the text. Hence the findings of the current study specifically reflect how readers refer to sources of the present documents. These references were not only mostly implicit but also infrequently included mentions of source characteristics, such as position, perspective, expertise, and motivation. Thus, our findings suggest that although most readers do construct some source representations while reading, these representations might not be sufficiently elaborate to support deep processing of information sources.

How can laypeople evaluate experts?

A possible interpretation of our findings is that the authors’ expertise might have conferred high reliability to the information sources and that this may have negatively impacted learners’ tendencies to explicitly identify source information and to evaluate source reliability. When laypeople read documents written by experts, they may reasonably assume the authors to be relatively trustworthy and therefore might be less epistemically vigilant in their assessments of expert sources (cf. Sperber et al. 2010). However, authors’ expertise does not make sourcing redundant. On the contrary, in controversial contexts people typically encounter expert disagreement and may need to pay attention to differences in source backgrounds, motives, qualifications, and competence in order to understand the causes of disagreement and to reconcile conflicts between experts (Bromme et al. 2015; Thomm et al. 2015). Thus, our findings suggest that low epistemic vigilance regarding expert sources may exacerbate the challenge of dealing with expert disagreement.

On the positive side, it is interesting to note that participants who did attend to source characteristics paid relatively high attention to authors’ positions and perspectives, along with their expertise. This suggests that when reading conflicting expert sources, identification of experts’ viewpoints may provide learners with an accessible heuristic for comparing and assessing expert authors, beyond their face qualifications. Awareness of experts’ viewpoints may help readers understand the reasons for differences between experts and the limitations of their claims.

Readers’ views of themselves as sources

If texts can be viewed as people who are involved in a social exchange of ideas (Britt et al. 2013; Wineburg 1991), then our data suggests that readers may view themselves as part of the conversation. The participants in our study did not only construct representations of the document sources while reading, they also constructed metacognitive representations of themselves as sources of knowledge on the topic: They reflected on the adequacy and origins of their own knowledge and compared and contrasted their personal knowledge, which was based on previous testimonial sources as well as first-hand experiences, with the knowledge claims put forward by the authors.

Effective processing of texts requires evoking and applying prior knowledge (Kintsch 1988). Yet our findings suggests that over-reliance on the reader’s knowledge for content evaluation may also create negative trade-offs in terms of sourcing. This relation could possibly work in both directions: Low sourcing abilities might cause readers to rely more heavily on their prior knowledge when evaluating what they are reading. The negative relation between topic interest and reliance on the reader as a source of knowledge suggests that this sourcing approach may also reflect low motivation for reasoning about the dilemma. This is not to say that effective sourcing entails a disregard for the reader’s knowledge. The high sourcers in our study also reflected on themselves as sources of knowledge, but they relied on their personal knowledge to a lesser degree and devoted more attention to source evaluation. This might be a more adaptive reading strategy in complex contexts in which laypeople typically have low prior knowledge and are limited in their abilities to engage in first-hand examination of the veracity of knowledge claims (Stadtler and Bromme 2014; Bromme and Goldman 2014). The ways in which learners position themselves as sources of knowledge in relation to testimonial sources may impact their sourcing practices and consequently their multiple document comprehension.

Epistemic perspectives and spontaneous sourcing

The value that learners assign to external and internal sources of knowledge may reflect their views of knowledge and knowing. Our findings revealed a marginally significant positive relation between absolutism and spontaneous sourcing. This trend parallels the finding of Bråten et al. (2014b) that an emphasis on justification by authority is positively correlated with source citations in student essays. From an absolutist perspective, experts can be a reliable sources of knowledge, but might also sometimes be mistaken or biased (Kuhn and Weinstock 2002; Kuhn 1991). Therefore, readers who endorse absolutist views might engage in sourcing primarily in order to ascertain the expertise, impartiality, and reliability of authors. In the present study, absolutism was not related to attention to specific source characteristics. However, it should be noted that spontaneous attention to source characteristics was very low in the current study and this might explain why clearer relations between absolutism and mentions of specific source features did not emerge.

Higher endorsement of multiplist views that knowledge is uncertain and that the sources of knowledge and justification are personal were found, in this study, to be related to lower levels of sourcing and to a greater focus on the reader as a source of knowledge. These results are in line with previous findings that views of knowledge and justification as personal are related to higher reliance on one’s own opinion for document evaluation (Strømsø et al. 2011). The results also support and explain a previous finding that multiplism is related to lower comprehension of author viewpoints (Barzilai and Eshet-Alkalai 2015). From a multiplist view, the goal of the task might be perceived as confirming personal beliefs or as forming a personal opinion. Therefore, standards of content evaluation might be primarily based on the extent to which the document coheres with the reader’s prior knowledge, rather than on assessment of external sources of knowledge.

Evaluativism was found to be positively related to references to the reader as a source of knowledge. However, attention to the reader as a source did not coincide, in this case, with lower attentiveness to document sources, suggesting that an evaluativist approach may indeed entail better balance of subjective and objective dimensions of evaluation. Yet, similarly to the findings of Barzilai and Zohar (2012), evaluativism was not related to higher sourcing activity. As Barzilai and Zohar (2012) have suggested, evaluativism might not be reflected in the amount of sourcing but rather in the ways in which learners think about sources. However, because spontaneous attention to source characteristics was quite low in the present study, it might not have been possible to discern such differences.

The development of epistemic thinking is marked by increased flexibility and greater adaptivity to task contexts and demands (Bromme et al. 2010b; Elby and Hammer 2001; Bromme et al. 2008). Indeed, evaluativist thinking has been found to be more highly adaptive to contextual cues (Barzilai and Zohar 2012; Barzilai and Eshet-Alkalai 2015). Therefore learners who endorse evaluativist views might not necessarily engage in effortful evaluation processes if these are not perceived to be imperative for task performance. It is possible that the conditions of the present study did not sufficiently alert participants to the importance of sourcing. Hence in order to better understand the role of evaluativism in sourcing it will be necessary to explore and compare a wider range of task conditions, for example, to compare prompted and unprompted sourcing. Finally, it is also possible that the measure we employed to assess epistemic thinking might not have been sensitive enough to capture differences in spontaneous evaluativist sourcing. This possibility will need to be examined by employing the epistemic thinking assessment in additional types of sourcing tasks in order to further test its predictive ability.

The contribution of sourcing to written argumentation

The findings of this study offer additional confirmation that higher levels of sourcing are related to construction of more complex and well-justified arguments that are based on multiple information sources. Readers who pay more attention to sources as they read and to the relations between sources and their claims may develop a more complex and well-integrated understanding of the controversial nature of the topic and therefore might tend to construct arguments that consider multiple sides of the problem. The correlation between source–source links and argument structure, as well as the analysis of Dor’s case, suggest that source–source comparisons may particularly highlight differences between sources that are subsequently reflected in the consideration of qualifications and counter-arguments.

Attention to the reader as a source of knowledge was negatively related to the number of reasons and information sources in the argument, but was not related to argument structure. A possible explanation is that the more readers rely on themselves as sources of knowledge the less likely they are to perceive a need for providing detailed justifications for their claims and for grounding these justifications in testimonial sources. Furthermore, greater reliance on the reader as a source of knowledge may lead to lower attention to document sources and hence to lower integration of documents in participants’ arguments.

Limitations and future directions

The think-aloud method employed in the current study has both advantages and drawbacks. On the one hand, this method provides detailed documentation of concurrent thinking processes, yet on the other hand, this is an obtrusive procedure that can slow down thinking processes and that may not be well-suited for assessing highly automated or highly effortful processes (Veenman 2011). Eye-tracking evidence indicates that some sourcing processes occur implicitly and automatically (Mason et al. 2014b). Hence, the levels of explicit sourcing documented in this study, and in similar think-aloud studies, might not reflect the full scope of participants’ sourcing processes. Additional methods of assessing spontaneous sourcing, such as eye-tracking procedures or examination of sourcing during collaborative learning, might be used to complement think-aloud protocols.

Several additional limitations should also be considered in the interpretation of the results of this study. First, the sample size of this study is not sufficiently large to yield strong conclusions regarding differences in sourcing practices. However, insights from this exploration may inform the design of future studies that might be conducted with larger samples. Second, we employed designed materials and therefore sacrificed some of the authenticity of Internet searching. This might have had an effect on learners’ sourcing approaches. In the future, it might be worthwhile to examine these issues in a more authentic setting. Third, the study topic was not chosen by the participants and was perceived by them as only moderately interesting. Participants’ motivation could have influenced their sourcing. Hence, it might be fruitful to study these issues using topics that are chosen by participants. Fourth, we did not control participants’ prior beliefs and these may have impacted how they processed belief compatible and incompatible texts. Future studies might explore how prior beliefs impact sourcing. Fifth, we did not compare expert sources to non-expert sources and therefore cannot draw definitive conclusions about the impact of authors’ expertise. Such a comparison awaits further research. Sixth, our topic knowledge measure was relatively brief and simple. Therefore, in the future, it might be advisable to expand this measure and improve its reliability.

An additional topic that might be explored in future research is the unexpected role of gender as a predictor of sourcing. Two prior studies have found women to be less successful than men in multiple document comprehension (Strømsø et al. 2008; Bråten and Strømsø 2010). In those studies, gender differences were attributed to differences in prior knowledge. However, in the present study, no gender differences in topic knowledge or topic interest were found, raising a question regarding the sources of these differences.

Instructional implications

There is growing scholarly interest in developing and testing models for fostering learners’ source evaluations (e.g., Britt and Aglinskas 2002; Wiley et al. 2009; Braasch et al. 2013; Mason et al. 2014a; Macedo-Rouet et al. 2013; Walraven et al. 2013; Kammerer et al. 2015). These studies generally provide encouraging evidence regarding the efficacy of instructional interventions for improving sourcing strategies. However, it is not yet clear to what extent learners are able or inclined to transfer these strategies and apply them in additional contexts, especially out-of-school contexts. The finding of this study suggest that learners can come to instruction with knowledge, skills, and dispositions regarding sourcing that have developed over extended use of the Internet for information seeking. Instruction that pays attention to students’ existing sourcing practices and builds on and expands these practices may enable more meaningful and robust learning that can extend to future contexts.

Instructional interventions might also benefit from attending to differences in the ways in which learners engage in sourcing. These may stem, in part, from learners’ views regarding the nature of sources, from their understandings of the aims, processes, and limits of knowledge construction, and from their perceptions of their own position vis-a-vis the sources they are reading. As a result of these differences, learners may experience and benefit differentially from sourcing instruction. Paying attention to learners’ epistemic views might therefore be highly relevant for sourcing instruction. In particular, the findings of this study suggest that learners who view knowledge construction as subjective and personal might find it more difficult to shift from a focus on “first-hand evaluation” to coordination of “second-hand evaluation” and “first-hand evaluation”. Instruction that explicitly addresses the importance and the challenges of integrating “first-hand” and “second-hand” evaluation, and that fosters both “first-hand” and “second-hand” evaluation strategies, may be beneficial for all learners, and especially for learners who tend to adopt subjective approaches to knowing.

Finally, our findings suggest that one of the main challenges of sourcing instruction is not just to increase the prevalence of sourcing but also to improve the quality and complexity of learners’ source representations. Learners may have limited understanding of the compound factors that underlie source reliability. This may be especially true when it comes to understanding the reliability of expert sources. Thus instructional interventions might need to pay greater attention to developing and expanding learners’ understandings of how expert knowledge is constructed, justified, and communicated.