Post-secondary education through e-learning platforms continues to grow, not only for distance education, but also combined with face-to-face activities in mixed or blended higher education courses, in both industrialized and developing countries (Seaman et al. 2018; Palvia et al. 2018). In the United States, the U.S. Department of Education’s National Center for Educational Statistics (NCES) Integrated Postsecondary Education Data System (IPEDS) database registered that 31.6% of all students were taking at least one online course (Seaman et al. 2018). Consequently, there is an increased interest in research addressing factors affecting e-learning outcomes. This paper is focused in the case of digital expository text comprehension within an e-learning environment.

Digital text comprehension

Reading comprehension today is a complex construct, defined as “the joint outcome of three sources of influence: the reader, the text and the activity, task or purpose for reading” (OECD 2019, p.30; see also Salmerón et al. 2018). Reader, text and task dimensions interact within a specific setting or scenario. So then, controlled or laboratory studies of text comprehension have moved away from “traditional, decontextualized assessment” (OECD 2019) adopting a scenario-based assessment approach. For digital text comprehension, these scenarios require participants to solve different tasks in hypertextual, mock Internet environments. Participants are asked to search and identify, make sense, integrate, compare, and evaluate information, navigating through multiple pages or screens (e.g. Coiro 2011a, 2011b; Leu et al. 2015; OECD 2009, 2019; Salmerón et al. 2018). This complex comprehension construct would be a particular case of what is termed multimedia learning (e.g. Clark and Mayer 2016) or student-centered learning environments (e.g. Azevedo et al. 2013) from other research perspectives.

Comprehension of digital dynamic texts depends partially on the same factors affecting printed or static texts such as general verbal ability and previous domain knowledge (Amadieu and Salmerón 2014; Coiro 2011a; Leu et al. 2015; Naumann and Salmerón 2016; OECD 2011). Vocabulary and previous domain knowledge allow students to read fluently and accurately, and to activate semantic knowledge needed to build an integrated model of what the text is about, in both print and digital reading tasks. For example, in the large multinational PISA study, the overall correlation between digital and print reading performance was .83 (OECD 2011, p. 74).

Another contributing factor is working memory, the neurocognitive system for temporary retention and processing, limited in capacity (Baddeley 2010). In traditional printed reading, individual differences in working memory have been associated with differences in text comprehension because working memory space constrains inferential processing and model construction (Peng et al. 2018).). In hypertext comprehension, more demanding navigation requirements affect low working memory participants in particular (Burin et al. 2015; Lee and Tedder 2003).

On the other hand, digital texts present additional specific features needing processing, over and beyond traditional printed texts. Readers have to navigate several information nodes, evaluating and selecting or discarding content, and integrating multimedia information from multiple sources (Coiro 2011a, 2011b; Leu et al. 2015; OECD 2011, 2019; Salmerón et al. 2018, Ben-Yehudah et al. 2018). Navigation, integration and evaluation emerge as critical skills for comprehension, and interact with traditional reading skills (Hahnel et al. 2016; Naumann and Salmerón 2016; Salmerón et al. 2018). For example, in an extension of the large PISA study in Germany, Hahnel et al. (2016) assessed the effects of linear reading, basic computer skills, navigation skills, and online information evaluation, on digital texts comprehension. They found that digital reading comprehension was predicted by linear reading and digital skills, which had both direct and interacting influences on comprehension outcomes.

Digital text comprehension in e-learning

E-learning presents a special case of a digital text scenario. To deliver expository instruction, thematic material is presented in text or video based lessons embedded in a closed learning content management system, such as Moodle, Blackboard, or similar (Clark and Mayer 2016). A typical course presents information in several lessons, spread across pages to be read (or videos, generally also with textual information), followed by assessments, implemented as multiple choice, open ended questions, or problem solving activities. As a digital text comprehension task, students have to navigate and integrate information presented along multiple pages, and also need to understand and navigate the learning environment, to adequately solve the comprehension and learning tasks.

On the other hand, e-learning differs from other digital text scenarios. One of the main differences stems from its conception as a self-learning device. In contrast to classroom or controlled settings, including experimental and large scale studies on digital text (OECD 2011, 2019; Salmerón et al. 2018, Ben-Yehudah et al. 2018), e-learning generally lacks a teacher or researcher to explain, model, guide and monitor behavior, relying more on self-generated strategies, timing and motivation (Azevedo et al. 2013; Broadbent and Poon 2015).

Another difference is that students are generally not required to search, navigate and evaluate different Internet pages or apps. Courses are self-contained, in the sense that the information needed for the learning objectives is presented within the course. Navigation is generally structured, with detailed instructions and steps. Source evaluation is not required, since the course is assumed to provide valid knowledge (at least for course success). However, e-learning is delivered online, and students complete the tasks in their home or habitual place of study, which means that students can do external searches, as well as other concurrent activities, again suggesting that self-regulation is an important aspect of comprehension in e-learning.

Self-regulation and metacognition in text comprehension

The ability to regulate behavior towards a goal, termed self-regulation, encompasses a wide range of cognitive, motivational / emotional and behavioral aspects, and has been studied under different models and perspectives. Across the variety of definitions and models, the metacognitive dimensions of planning, monitoring and strategic activities come across as its core (Azevedo et al. 2013; Graesser and McNamara 2010; Panadero 2017; Pintrich 2000; Winne and Hadwin 1998, 2008; Zimmerman 2008). Highly self-regulated students set up learning goals, plan and guide their activity according to this plan, monitor their progress towards the goals, even reassessing or changing plans and goals, and adopt strategic goal directed behaviors. Models of self-regulated learning emphasize that these metacognitive processes are dynamic and interlaced with motivational and affective processes, and also involve judgments of knowledge and enjoyment (Azevedo et al. 2013; Panadero 2017; Winne and Hadwin 2008; Zimmerman 2008). For example, Winne and Hadwin (1998, 2008) which has been influential in computer supported learning settings (Azevedo et al. 2013; Panadero et al. 2015), views self-regulated students as active, goal directed, who manage their own learning via monitoring and the use of metacognitive processes, while recognizing the interplay between self-regulatory actions and motivation (Winne and Hadwin 2008). The model describes how students constantly monitor their activities against standards and employ different strategies to perform tasks.

In the reading comprehension literature, metacognition encompasses setting a reading task goal and planning towards it, ongoing monitoring to adapt reading behavior, and enacting strategies to support comprehension (Afflerbach et al. 2017; Cromley and Azevedo 2011). The metacognitive regulation of reading is expressed as reading strategies, “deliberate, goal directed attempts to control and modify the reader’s effort to decode text, understand words, and construct meanings of text” (Afflerbach et al. 2017, p. 38). A metacognitive strategy is defined as an intentional and conscious activity towards understanding; they are activities deployed during a reading task.

Strategic processes in reading have been considered cognitive or metacognitive, a distinction which varies according to models and authors (Cromley and Azevedo 2011; Veenman 2011). Following Veenman (2011), higher-order metacognitive processes plan, monitor and regulate lower-order cognitive processes that, in turn, shape behavior. In models of text comprehension, the cognitive processes encompass lower level decoding, lexical activation, sentence processing, inferential activity, and model building; these might be automatic cognitive skills, or might need specific cognitive strategic processing (Graesser 2007). In Veenman’s (2011) example, combined with Graesser’s (2007) view, drawing inferences is a cognitive activity required for comprehension; it might be automatized, or the focus of a particular strategic behavior, but the decision to initiate such strategic behavior is a metacognitive one.

Metacognitive strategic activities should be distinguished from other meanings of metacognition, for example, metacognitive knowledge, or metacognitive evaluation. Metacognitive knowledge taps whether students know, in a general way, which strategies are more effective for a particular reading goal. For example, in the PISA assessment (2009, 2011) students were asked to evaluate the effectiveness of different strategies (presented in multiple choice format) for two hypothetical tasks of summarizing information, and understanding and remembering a text. Metacognitive evaluations consist of judgments about task or text difficulty, his/her own learning, or feelings of knowing. In the Azevedo et al. (2013) model, they are superordinate self-regulating dimensions, such as students’ ongoing awareness and judgments about his/her domain knowledge, expectations of results, and learning goals; and guide metacognitive activities as described before.

Studies on metacognition in digital comprehension

In laboratory or classroom environments, metacognition has been shown to play a relevant role in digital text comprehension, in studies with children, adolescents, and college aged participants with different methodological approaches, such as think-aloud protocols, instruction in metacognition, and a special category of intervention, intelligent tutoring systems. Also, these perspectives can be blended; for example, the EARLI-Centre for Innovative Research (E-CIR) initiative seeks to combine multimodal indexes of self-regulation such as log-files, eye tracking, facial expressions of emotion, heart rate and electro-dermal activity when learning from digital platforms (Bannert et al. 2017).

Studies with think aloud protocols typically ask students to verbalize their thoughts while navigating and responding questions about a digital text, which allows capturing the dynamic flow of metacognitive regulation (Afflerbach and Cho 2009; Cho and Afflerbach 2015; Cho et al. 2018; Coiro et al. 2008; Leu and Castek 2006; Moos and Azevedo 2008; New Literacies Research Team 2005). As an early example, Moos and Azevedo (2008) had students freely navigate Microsoft’s Encarta encyclopedia to learn about a topic (the circulatory system), registering verbal protocols. They found that verbalizations could be characterized in three dimensions, planning, monitoring, and strategy use, and that high previous domain knowledge was associated with more planning and monitoring.

For their part, experimental intervention generally assess the efficacy of an instructional program targeting one or more metacognitive process (de Vries et al. 2008; Fesel et al. 2016; Hagerman 2017; Kuiper et al. 2009; Kuo and Hwang 2014; Naumann et al. 2008; Salmerón and Llorens 2018). For example, Salmerón and Llorens (2018) found better digital comprehension with an experimental intervention on metacognition focused on planning a search, anticipating and evaluating links, monitoring, and prior knowledge activation. The intervention consisted of explanations and modeling by an instructor, followed by students’ practice, including students’ evaluations of taped videos about other students’ strategies.

Finally, the latter category of studies comprises interventions with “intelligent tutoring systems” or “pedagogical agents” that prompt students to engage in planning, monitoring and strategic behaviors, and respond and provide feedback along the learning session, seeking to scaffold the internalization of these metacognitive aspects (Betty’s Brain, Biswas et al. 2010; iSTART and SERT, McNamara and Magliano 2009; McNamara 2017; McCarthy et al. 2018; MetaTutor, Azevedo et al. 2010; SEEK, Graesser et al. 2007). For example, the SERT program (McNamara and Magliano 2009; McNamara 2017), specifically aimed at monitoring text comprehension and generation of self-explanations, showed benefits for low prior knowledge college students when reading scientific texts. However, other studies found that this type of tutors can overload cognitive resources, hindering comprehension outcomes (McCarthy et al. 2018).

A meta-analysis of intervention studies in metacognition (Lan et al. 2014), including both instruction studies and intelligent systems, found that the effects on comprehension were mixed, and mediated by type of text and reading abilities. It also suggested that motivation, quality of instruction, and peer interaction played relevant roles.

Self-report measures of metacognition in traditional text comprehension

In studies of traditional expository text comprehension, self-report questionnaire measures have been developed to measure the metacognitive skills students know and employ when reading (Cromley and Azevedo 2011; Mokhtari and Reichard 2002; OECD 2011, OECD 2019; Schellings and Van Hout-Wolters 2011). They are relatively cost-effective, quick and easy to administer and score, can be administered to large samples, and their psychometric properties can be examined. However, self-report measures of metacognitive strategies have been questioned: asking students what they generally do when learning, with a “broad brush” scope, might reflect students’ perception rather than strategies actually performed (Cromley and Azevedo 2011; McCardle and Hadwin 2015; Samuelstuen and Bråten 2007; Schellings and Van Hout-Wolters 2011). Instead, metacognitive questionnaires should be based both on theoretical dimensions and tailored to particular tasks or contexts (Cromley and Azevedo 2011; McCardle and Hadwin 2015; Samuelstuen and Bråten 2007; Schellings and Van Hout-Wolters 2011).

Participants have been shown to be relatively more accurate at reporting the strategies they are currently using or just employed following a task (Cromley and Azevedo 2011; Samuelstuen and Bråten 2007). Also, metacognitive strategies have obtained better predictive accuracy when referred to a particular reading task. In a series of studies, Cromley and Azevedo (2011) showed that measures of reading comprehension task-specific strategy use were related to comprehension outcomes, and also to vocabulary, inferential processing, and background knowledge, in high school and undergraduate participants. For their part, Samuelstuen and Bråten (2007) constructed a questionnaire in conjunction with a concrete reading task (reading an expository text about socialization). They compared a general measure of cross curricular metacognitive strategies (addressing “when I study…”) with a task-specific strategy inventory, in which items were adapted to refer to a reading task just finished as a frame of reference. Results in text comprehension were linked to the task specific measures, but not with the general one.

The present study: Self-reported metacognitive activity in digital text comprehension in e-learning.

To recap, evidence suggests that metacognitive planning, monitoring, and strategic behavior lead to better digital text comprehension, and also that metacognitive activity is associated with better performance, whereas the success of metacognitive training depends on previous knowledge and skills, and on the quality of intervention. Self-reported metacognitive measures have been associated with traditional expository text comprehension, provided that they refer to a specific reading task.

Studies on metacognition in digital text comprehension have been conducted in controlled laboratory or classroom testing environments. They are test-like situations with the presence of instructors who can guide and monitor performance, and with computers specially prepared for the tasks. In contrast, e-learning at college level is frequently carried out at the students’ home or habitual place of study, with their own computers, at their own pace and without a supervisor. These differences suggest that metacognitive self-regulation might be an important outcome predictor.

So then, the purpose of the present study has been to examine the contribution of self-reported metacognitive activity to expository digital text comprehension in an e-learning scenario. To that end, first year college students completed reading comprehension tasks in an e-learning “course”, especially designed for research purposes and thematically not related to their course, in their home or usual place of study. Participants read two digital texts (eight subnodes, with a lateral navigation menu and embedded hyperlinks), and answered literal and inferential questions about them. Upon completion, they reported their metacognitive activities during the comprehension tasks with the Metacognitive Inventory, an adaptation of the Metacognitive Awareness of Reading Strategies Inventory (MARSI, Mokhtari and Reichard 2002), referred to the reading tasks just completed.

Specifically focused on expository text comprehension for school purposes in upper elementary or middle school and beyond, Mokhtari and Reichard’s MARSI (2002; Mokhtari et al. 2018) is one of the most employed across many languages and cultural contexts (Mokhtari et al. 2018). The MARSI was designed to assess students’ awareness and perceived use of reading strategies. The inventory evaluates three strategic domains that can be mapped to the three metacognitive aspects of planning, monitoring, and strategic behavior described previously: global strategies, encompassing the planning dimension (e.g. having a reading purpose in mind, deciding what parts to read more carefully), problem solving strategies, tapping monitoring of difficulties and errors (e.g. re-reading or adjusting reading speed when difficulties arise), and support strategies including note taking, highlighting, underlining, and other concurrent activities. For this study, we selected a subset of items (related to the global and problem solving dimensions), and adapted them to the reading tasks presented in the e-learning platform.

Participants completed the reading comprehension tasks, the self-reported metacognitive activity in those tasks, and also answered a questionnaire about their Internet frequency use and experience. In another session they completed verbal ability and working memory tests (Vocabulary and Letter Number Sequencing) in the lab.

The following hypotheses guided the analyses:

H1. The Metacognitive Inventory will show adequate construct validity and reliability.

H2. Expository digital text comprehension in an e-learning scenario will be associated with verbal ability, working memory, and Internet experience.

H3. Expository digital text comprehension in an e-learning scenario will be associated with metacognitive self-reported activity.

Psychometric properties of the Metacognitive Inventory were analyzed in the first place; after that, we assessed the contribution of factorial scores in metacognitive domains, verbal ability, working memory, and Internet experience contribution to digital comprehension.

Method

Participants

Two hundred and twenty-eight first year undergraduate psychology students, enrolled at a public university, initially took part in the study for course credit. Nine participants were discarded: five did not complete the reading and comprehension tasks, and four took less than two minutes to answer one comprehension questionnaire. The final sample consisted of 219 participants, 168 women, 51 men, mean age = 22.79, SD = 6.44.

All participants filled an informed consent form, and the project was assessed and approved by an institutional ethics committee.

Materials

Text comprehension

Two expository texts about low previous knowledge content for the participants (Telescopes in Astronomy, Particle Physics) were employed (Burin et al. 2018). Texts were adapted from Wikipedia and online sources. They had a similar argumentative structure: main concept, two secondary concepts, relevant and irrelevant details about each, a conclusion relating both. Their length was, for Astronomy, 1684 words without titles, 1709 total words; for Physics: 1608 words without titles, 1635 total words.

Texts were analyzed with an automatized readability software (https://legible.es) which calculates the Huerta score, an adaptation of the Flesch Reading Ease score to Spanish (Fernández 1959). The Huerta index for the Particle Physics text was 55.82, and for the Telescopes in Astronomy text, 55.44; both corresponding to “somewhat difficult” and “pre-university level” qualitative categories.

Low previous knowledge texts were chosen to maximize comprehension variance. They were pre-tested and validated in a previous study about hypertext reading, with a similar first year psychology college sample (Burin et al. 2018), by comparing accuracy in comprehension scores with two high previous knowledge texts (about psychology topics, Human Memory and Language). Participants reading the Particle Physics and Telescopes in Astronomy hypertexts correctly answered significantly less comprehension questions than on Memory and Language hypertexts.

Texts were divided in eight nodes and presented in eight screens (one per node), with titles. Six screens also showed a decorative, irrelevant picture, in Wikipedia style, to increase ecological validity of the texts. Decorative images, such as images of the sky or Galileo in the Telescopes text, are common in online presentations.

Navigation could be accomplished by a hierarchical concept map in a navigation side bar, or by clicking two embedded links. When they finished reading, they had to click on a button labeled “Finished – Go to Questions”. This button brought the participant back to the Course page, where they clicked on the Questions. Figure 1 shows sample text pages. Text pages can be seen at the task demo video (https://osf.io/yt6s7/).

Fig. 1
figure 1

Sample text pages (cropped)

Full size image

Each text was followed by 10 multiple choice questions. Questions were designed to vary in difficulty, requiring either literal information, inferences connecting two sentences or paragraphs in the same node or screen, or inferences based on more than one paragraph and requiring integration and elaboration between two or more nodes or screens. These questions were employed and validated in a previous study ([Author]).

Metacognitive inventory (MCI)

A self-report measure of metacognitive regulation during the tasks was designed, based upon the MARSI’s local adaptation (Mokthari & Reichardt, 2002; Piacente 2003). MARSI’s Global and Problem Solving subscales, referred to the planning and monitoring metacognitive dimensions, were adapted to refer to the reading task just completed, following Bråten and Strømsø (2011) criteria for metacognitive self-report questionnaires, which emphasize close relationship to a specific reading task. Wording was changed to refer to the task just performed, and two items included looking at “titles, navigation bar”. Items from the Support subscale referring mainly to paper-based activities (e.g. write summaries, underline, use dictionaries), or that were not adaptable to the digital reading task, were discarded. The final version comprised 18 items. Another questionnaire about strategic activities for completing these particular digital reading tasks (such as taking notes in paper and pencil, or in a Word, text or Note document, navigating back and forth with the browser, taking screenshots, Googling answers) was added but not analyzed here.

Scoring asked whether participants had done what the item describes in one text, both, or none (ordinal scale). When students’ self –report their metacognitive activity through Likert scales they may choose various reference points for comparing their conceptions about their performance, e.g., their own individual standard, an ideal-good or ideal-bad student standard, or what they think a teacher might require (Schellings and Van Hout-Wolters 2011). A three-points-scale asking whether they did or did not apply a particular activity to a text (e.g.: “I previewed, read the text in general, and then read its parts”: Yes in both texts, Yes in one text, No) may then be used to reduce the variation in the choice of a reference point (Schellings and Van Hout-Wolters 2011).

Verbal ability

Participants completed the BAIRES (Cortada de Kohan 2004). This is a Vocabulary test with two subtests, Synonyms and Definitions, which has been standardized and validated for collective verbal ability assessment in our population.

Working memory

Participants completed an adaptation of the Letter-Number Sequencing subtest from the Wechsler Adult Intelligence Scale (Wechsler 2003). Scoring was computed as number correct, taking into account the discontinue rule, as described in the Manual (Wechsler 2003).

Internet experience

Participants were asked to categorize their time spent online in computer or tablet devices (excluding cell phones). The Internet Frequency categories were similar to those in the PISA 2009 study (OECD 2011), adding a category for very frequent users given the participants’ demography (young college students). Categories were: less than once a week; one to five hours per week, almost every day, up to an hour; and almost every day, several hours. This questionnaire showed moderate and significant correlations with the number of Internet activities and operational skills in a previous study (Burin et al. 2018).

A sample session video showing the digital materials is available at the Open Science Framework (https://osf.io/yt6s7/). Materials in their original language can be sent upon request.

Procedure

In the first session, participants completed the informed consent form and verbal ability tests. In a second online and remote session they completed general Internet experience questions, two texts (each followed by questions), and the MCI. A sample session video is available on the Open Science Framework (https://osf.io/yt6s7/).

Tasks were then implemented within courses (each study condition was a course, counterbalancing text order presentation) with Moodle 2.6 and hosted in an external institution server, reinforcing that they were part of a study and not related to their regular course performance.

Data analyses

Analyses were carried out with R 3.3.2 (R Core Team, 2016). In the first place we conducted a psychometric analysis of the MCI measure to assess its structure and obtained composite measures of metacognition using the packages psych 1.8.4 (Revelle 2016), lavaan 0.6–3 (Rosseel 2012), and semTools 0.5–1 (Jorgensen et al. 2018). To test the association between ability, metacognition, Internet experience, and comprehension, generalized linear mixed models were implemented with lme4 1.1–17 (Bates et al. 2015) and lmerTest 3.0–1 (Kuznetsova et al. 2017).

Digital text comprehension was computed in two ways: 1) as a global measure, to examine correlations with the other variables, 2) as individual items entered as dependent variables in a general linear mixed model. We employed two texts followed by 10 comprehension questions each (total dependent variable = 20 items) and counterbalanced their presentation, to have more statistical power and generalizability. Instead of computing total accuracy per subject, and then averaging across conditions, mixed models take into account the full data (e.g. all responses by item by subject), thus providing more statistical power. Mixed models account for participants’ and items’ extraneous sources of variation, entering them as random parameters, allowing to control for this variance in the fixed effects estimation (e.g. metacognition, verbal ability, working memory, Internet experience), and providing a statistical tool for generalization beyond these particular participants and items (Baayen et al. 2008).

Data and code to reproduce analyses are available at the Open Science Framework, data: https://osf.io/u2ne7/; R code: https://osf.io/9a7rh/.

Results

Metacognitive inventory

The MCI psychometric properties were analyzed first, following Revelle (2016), and model fit indexes suggested by Hu and Bentler (1999). To interpret the scree plot and determine the number of factors to retain, an exploratory factor analysis with the simulation-based Parallel Analysis with minres least squares extraction suggested three factors and three components. An exploratory factor analysis with three factors, minres extraction and Varimax rotation, showed an acceptable KMO value (MSA = 0.69) and had good fit indexes (TLI = 0.948, RMSEA = 0.025 [CI90 = 0–0.042]). The first factor accounted for 52% of the variance, the second, 25%, and the third, 23%. Table 1 shows item content, standardized factor loadings, and communalities, for this three-factor model. The first factor can be considered a global monitoring or regulation of reading dimension, including having in mind the task purpose, re-reading and paying attention to important or difficult parts. The second one seems to refer to text features or local problem solving, such as disorientation or lack of understanding, and the use of typography and navigation elements as comprehension aids. Two items load on the third one, referring to the organization of the reading task: previewing and then reading the parts, and looking at the whole text organization and length.

Table 1 Item content and standardized factor loadings for the three-factor model
Full size table

A confirmatory factor analysis for the three-factor model, where items with factorial loading ~ ≥ .30 were assigned for each latent factor (F1: Global/ Monitor, F2: Problem Solving, F3: Organization), showed good fit indexes (CFI = 0.934, TLI = 0.922, RMSEA = 0.029 [CI90 = 0–0.049], SRMR = 0.058). However, the Organization factor (two items: “Previewed text in general, read parts after” and “Paid attention to text organization and length”) had negative latent variable variance even when correcting the model with both items’ original variance. Therefore a two-factor model, excluding the Organization latent factor and its manifest variables, was tested. This two-factor model (Global Monitor, Problem Solving) showed better fit indexes (CFI = 0.981, TLI = 0.976, RMSEA = 0.017 [CI90 = 0–0.045], SRMR = 0.049). For comparison purposes we run a one-factor model with these items, which had significantly worse fit (χ2 Diff = 26.96, DF = 1, p < .00001) and had worse fit indexes, CFI = 0.863, TLI = 0.836, RMSEA = 0.046 [CI90 = 0.024–0.065], SRMR = 0.061, AIC 2factor = 6536.6, AIC 1factor = 6561.6.

Table 2 shows the parameter estimates for items’ factorial loadings for the two-factor solution.

Table 2 Parameter estimates for items’ factorial loadings in 2-factor model
Full size table

Internal consistency was in the acceptable range for research purposes for the Global / Monitor factor, Cronbach’s α = .68 and McDonald’s ω = .68, similar in range for the total scale, Cronbach’s α = .64 and McDonald’s ω = .67, but lower for the Problem Solving factor, Cronbach’s α = .46 and McDonald’s ω = .46.

For further analyses, each participant’s standardized factorial scores in both latent factors were computed.

Relationship between verbal ability, working memory, internet experience, metacognition, and digital text comprehension

Table 3 shows descriptive statistics for verbal ability, working memory, Internet experience, metacognition, and digital text comprehension. Total verbal ability (Vocabulary score) and working memory (Letter Number Sequencing score) were computed, and expressed as standardized scores. Internal consistency was in the acceptable range for the Letter Number Sequencing, Cronbach’s α = .79, and Vocabulary, Cronbach’s α = .70. Total digital text comprehension scores were computed, and expressed as z scores. Internal consistency was in the acceptable range, Cronbach’s α = .67. Metacognition comprises standardized factorial scores in both latent factors from the previous analyses (for the Global / Monitor factor, Cronbach’s α = .68, for the Problem Solving factor, Cronbach’s α = .46).

Table 3 Descriptive statistics for Verbal Ability, Working Memory, Internet Experience, Metacognition, and Digital Text Comprehension
Full size table

Given the presence of extreme outlier values (−2.5 ≤ z ≥ 2.5) in working memory and verbal ability, these variables’ distribution were winsorized, e.g. trimmed by 0.01 (winsor function, Revelle 2016) for further analyses.

Correlations between verbal ability, working memory, internet experience, metacognition, and digital text comprehension are shown in Table 4. Both cognitive ability measures, verbal ability and working memory, are significantly correlated between them. Also, both metacognitive measures are moderately and significantly correlated between them. Internet experience has a small and significant correlation with verbal ability. Finally, overall score in online text comprehension shows small and significant correlations with Internet experience and Global Monitor metacognitive scores, and moderate and significant correlations with verbal ability.

Table 4 Correlations between Verbal Ability, Working Memory, Internet Experience, Metacognition, and Digital Text Comprehension
Full size table

Contribution of verbal ability, internet experience and metacognition to digital text comprehension

To test the contribution of verbal ability, working memory, Internet experience, and metacognition to digital text comprehension, they were entered as predictors in a binomial generalized linear mixed model (glmer with logit link and maximum likelihood), with all individual item response accuracies as dependent variables. Participants and items were entered as random intercepts. The model showed that students’ comprehension was significantly and positively predicted by verbal ability (Estimate = 0.37, SE = .06, z = 6.25, p < .0001), Global / Monitor metacognition (Estimate = 0.54, SE = .16. z = 3.33, p = .0008), and Internet experience (Estimate = 0.11, SE = .05. z = 2.42, p = .0162), but not by working memory (Estimate = 0.01, SE = .05, z = 0.19, ns). Table 5 shows the summary for the model with random and fixed factors. Participants with more verbal ability, more metacognitive skills, and more Internet experience were more likely to correctly answer a comprehension question. Neither working memory nor problem solving metacognitive scores were significantly related to comprehension.

Table 5 Model summary for random and fixed factors
Full size table

Discussion

In summary, this study examined the contribution of self-reported metacognitive activity when reading expository digital text comprehension in an e-learning environment, in first year college students. Participants completed a questionnaire about their Internet frequency use and experience, read two digital texts, and answered questions about them, in their home or usual place of study. After that, they reported metacognitive activities during the comprehension tasks with the Metacognitive Inventory. They also completed verbal ability and working memory tests in a lab session.

The metacognitive inventory

Our first hypothesis concerned the Metacognitive Inventory’s psychometric properties, factorial validity and reliability.

Exploratory factor analyses suggested three factors: a global monitoring regulation of reading dimension, including having in mind the task purpose, re-reading and paying attention to important or difficult parts; a local problem solving factor, comprising actions in disorientation or lack of understanding, and the use of typography and navigation elements as comprehension aids; and a third dimension referring to the organization of the reading task, with two items, previewing and then reading the parts, and looking at the whole text’s organization and length. Confirmatory factor analyses showed metric problems for the third factor. A two-factor model, excluding the organization latent factor and its manifest variables (Global/Monitor, Problem Solving) showed good fit indexes. In correlational analyses, participants’ factorial scores in both metacognitive dimensions were moderately and significantly correlated between them. However, in confirmatory factor analyses a one-factor solution did not have adequate fit indexes, suggesting their separation.

Items selected for the adaptation came from two MARSI subscales, Global and Problem Solving (Mokhtari and Reichard 2002; Mokhtari et al. 2018). Therefore, this study replicates their structure, adding evidence supporting this proposed organization of self-reported metacognitive activities in reading. In this case, as different from the MARSI, the items asked if the actions described were performed in a digital expository text reading task, instead of a more general hypothetical reading situation.

In sum, one contribution of the present research is the construction and validation of a self-report measure for metacognitive activity in a digital comprehension task. Our results replicate the finding of two dimensions of metacognitive activity, global/monitor and problem solving, and extend these to an actual reading task, and to the digital comprehension domain. The Metacognitive Inventory can provide a psychometric cost-effective measure of metacognition in research and applied settings.

One limitation of the measure stems from the internal consistency indexes, which were in the acceptable range for research purposes (Nunnally 1978) for the Global / Monitor factor, but low for the Problem Solving factor. This low reliability also questions the construct validity of this dimension, and constrains further conclusions regarding this factor. Future research should examine if these results extend to other samples, and possible scale modifications.

Also, future research can combine the self-report measures with performance indexes such as navigation indexes examining log files, or eye-tracking. It should be taken into account that this study was interested in an e-learning scenario. Students completed the tasks at their home; they were advised to set up the time to complete the tasks in one session, but in effect they could start and stop, go grab a coffee or be interrupted by other factors, do other things simultaneously, as one would expect in such a naturalistic setting. Analyses of log files, as well as other behavioral measures, would be better suited for laboratory controlled studies.

Verbal ability and internet experience contribute to digital text comprehension in e-learning

Our second working hypothesis asked whether known factors affecting digital text comprehension were at play in this e-learning scenario. We found that comprehension was significantly correlated with Internet experience and verbal ability. The same pattern of results was obtained with the linear mixed model: students with more verbal ability, and more Internet experience were more likely to correctly answer a comprehension question. In contrast, working memory capacity was not significantly correlated with comprehension, and individual differences in working memory did not significantly contribute to comprehension in the linear mixed model.

This pattern of results is in line with previous studies of digital text comprehension showing strong effects of verbal ability, or offline reading comprehension abilities (Coiro 2011a; Hahnel et al. 2016; Leu et al. 2015; Salmerón et al. 2018; OECD 2011). Results also align with studies showing that digital text comprehension is related to Internet experience, or operational and navigational Internet skills (Burin et al. 2018; Hahnel et al. 2016; Salmerón et al. 2018). The relevance of this study is that these results are replicated an extended to an e-learning setting.

In contrast, working memory capacity was not associated with comprehension. This finding should be taken into account when trying to generalize controlled studies to e-learning settings, and deserves further research. It could be argued that in this scenario, where students were completing the task at their own pace, in their home, and without supervision, they could be implementing support strategies to circumvent mental load problems. For example, they could be taking notes, or taking breaks when they felt overloaded, or other individual strategies not captured in more constrained settings.

Metacognition contributes to digital text comprehension in e-learning

Finally, the third working hypothesis concerned the relationship between metacognition and comprehension. Global/ Monitor metacognitive scores were significantly correlated with overall score in online text comprehension, and in the linear mixed model, students with more Global/Monitor metacognitive activity were more likely to correctly answer a comprehension question. Results concerning the Problem solving dimension are questionable because of the low reliability of the subscale.

This association of self-reported metacognitive activity and digital text comprehension in an e-learning scenario aligns with a body of research relating self-reported metacognitive activity with traditional printed reading comprehension, as well as previous laboratory or classroom studies of digital reading comprehension with other methodological approaches such as verbal protocols.

The metacognitive measures were not correlated significantly either with cognitive ability or Internet experience. These results suggest that better performance associated with metacognitive activity might be partially independent from cognitive skills. On the other hand, intervention studies have shown that cognitive skills, for example reading skills or working memory capacity, can be a limiting factor when acquiring metacognitive strategies (Lan et al. 2014; McCarthy et al. 2018). In this case we studied self-reported metacognitive activity, already established, as different from those studies. Future research can shed further light on the interplay between metacognitive activity, cognitive abilities, and Internet skills in the e-learning scenario.

In a broader sense, these results support the idea that self-regulatory skills are an integral part of e-learning. With regard to self-regulation models (e.g. Azevedo et al. 2013; Panadero 2017; Winne and Hadwin 2008), this study has addressed a particular dimension of metacognitive activity, as different from metacognitive knowledge or metacognitive evaluation, general goal setting, or motivation. Future research could analyze how these findings are accommodated with other self-regulatory dimensions.

Finally, given the interest in a naturalistic e-learning scenario, these findings would be relevant for applied instructional and educational settings. Even when students can work at their own pace, in their own computers, and can search online what they need, solving digital comprehension tasks in such an environment requires verbal ability, Internet skills, and active metacognitive strategies. These factors should be taken into account when designing an e-learning course.

Given the expansion of online learning, future research should extend these results for other age populations, non-academic environments, or different methods of content delivery, such as videos or games.