Introduction

Poor student engagement has been recognized as a major problem observed in college classrooms (Martin 2009). Low engagement in academic reading is a typical example. Reading in college often involves conceptually complex and sophisticated texts that demand deep-level processing by students. However, many do not seem to apply much effort in reading course materials. They often invest an insufficient amount of time in reading and attend class without having read their textbooks (Arquette 2010; Phillips and Phillips 2007). Even among students who read their textbooks, many demonstrate a superficial level of reading by skimming the texts and using low-level reading strategies (Elias 2005; Lesley et al. 2007; Taraban et al. 2000). These adverse reading behaviors imply that many college students do not enjoy reading course materials. Students often report negative affects toward academic reading such as displeasure and boredom (Brost and Bradley 2006; Fitzpatrick and McConnell 2009; Lesley et al. 2007).

In response to low reading engagement among college learners, researchers developed a web-based tutoring environment, the virtual tutee system (VTS), that places students in the role of tutor (Park and Kim 2012, 2014). Existing research has repeatedly shown that student tutors exhibit a deep level of engagement and enhance performance after teaching their peers (Arco-Tirado et al. 2011; Benware and Deci 1984; Cushing and Kennedy 1997). However, implementation of peer tutoring in a college class appears to be constrained in many ways. Many college courses are lecture-based with a large number of students enrolled, with the classes meeting twice or three times a week at most for a very limited time. This arrangement of college courses hinders instructors from creating and overseeing a number of peer tutoring groups and also inhibits students from building rapport and meaningful interactions within the groups. On the other hand, the virtual tutee system (VTS) walks students through tutoring activities, thereby minimizing the extensive effort and time needed to learn how to teach their tutees. Thus, the VTS has been proposed as an alternative way to enable students to benefit from the experience of teaching peers in a college environment. The VTS is designed to enable students to experience deep engagement in reading and broad understanding of the content.

The VTS has been refined through two iterations of field trials (Park and Kim 2014). Evidence emerged in the field trials indicating that the VTS improved students’ reading engagement. For example, after using the VTS, students reported thorough reading and their use of reading strategies increased. The purpose of this study is to examine the effectiveness of the VTS for enhancing students’ reading motivation and reading engagement, which will lead to successful content learning. In the following, we illustrate the underlying theories of peer-tutoring effects (or, learning-by-teaching) and describe how these theories informed the design of the VTS.

Theoretical foundations

The literature on peer tutoring has consistently reported that student tutors benefit from teaching experiences. Student tutors’ knowledge and skills improved (Fisher 2001; Tang et al. 2004). In addition, student tutors demonstrate enhanced motivation and engagement. For example, students who taught their peers reported greater intrinsic motivation and higher self-efficacy beliefs than those who studied for their own learning (Benware and Deci 1984; Miller et al. 2010). Also, student tutors showed better class participation (behavioral engagement), spent more time on conceptual understanding of the subject matter (cognitive engagement), and reported higher interest in the content (emotional engagement) (Arco-Tirado et al. 2011; Benware and Deci 1984; Galbraith and Winterbottom 2011). This tutor learning effect is often referred to as learning-by-teaching (Gartner et al. 1971).

The underlying mechanism of learning-by-teaching can be explained by self-determination theory (SDT). According to SDT (Deci and Ryan 1985), individuals are more likely to exhibit high quality motivation and engagement when the learning environment is arranged in a way that supports the basic psychological needs for autonomy, competence, and relatedness, that is, when people experience a sense of psychological freedom (autonomy), a feeling of being competent (competence), and emotional bonds with others (relatedness). Peer tutoring inherently creates a condition that satisfies these three psychological needs. When students are put in the role of tutor, they tend to take on characteristics similar to the role of the teacher (Allen and Feldman 1973; Hogg and Vaughan 2005). The role of teacher usually implies independence and authority with a perception of having the capability to make decisions and help others (Allen and Feldman 1976). Such characteristics foster the perception of autonomy in student tutors. Also, student tutors develop a feeling of competence by being recognized as those who can teach others. In addition, tutor-tutee interactions enable students to establish social relationships, which satisfies a need for relatedness. Empirical support has also shown that students demonstrate an increase in intrinsic interest, self-confidence, and feelings of relatedness after tutoring their peers (Benware and Deci 1984; Fantuzzo et al. 1995; Miller et al. 2010). Thus, by serving as peer tutors, students can experience fulfillment of the three basic psychological needs and show deep motivation and engagement in learning.

Although peer tutoring fosters a learning environment with inherent support for these three basic needs, prior studies have indicated that the learning-by-teaching effect can be undermined under certain tutoring circumstances. When student tutors do not see the value of a task and/or when they value only extrinsic goals such as wealth and fame rather than intrinsic goals such as personal growth, they may not be able to benefit from learning-by-teaching effects even after peer tutoring. A minimal effect of learning-by-teaching was found among those who had negative motivational beliefs such as low task interest (Roscoe 2008). SDT describes self-endorsed value and personal goal as two additional sources of autonomous motivation and active engagement (Deci and Ryan 1985). The theory explains that the lack of self-endorsed value and pursuit of extrinsic goals compromises the feeling of autonomy and competence (Deci and Ryan 2000). Thus, a peer-tutoring environment should foster student tutors’ perception of task value and pursuit of intrinsic goals in order to adequately produce learning-by-teaching effects.

In summary, peer tutoring affords the inherent fulfillment of the three needs for autonomy, competence, and relatedness in student tutors, which can promote their autonomous motivation and active engagement. In order to maximize learning-by-teaching, a peer tutoring environment should be arranged not only to implement satisfaction of the three needs but also to provide support that allows student tutors’ perception of task value and pursuit of intrinsic goals. The design of the VTS focused on embodying this theoretical framework to develop a virtual tutoring environment that could promote student tutors’ engagement (see Fig. 1).

Fig. 1
figure 1

Theoretical framework guiding design of the VTS

Full size image

Design of the virtual tutee system

The virtual tutee system (VTS) is a web-based tutoring environment in which students teach a virtual character about the course materials they have been assigned to read (see Fig. 2). Based on the theoretical framework illustrated in Fig. 1, four design principles were proposed to guide development of the VTS (for more details, see Park and Kim 2012). The essential design features of the VTS include a simulation of the human tutoring environment and focuses on securing support for the three psychological needs for autonomy, competence, and relatedness that human tutors would experience in human peer tutoring. Also, the design of the VTS incorporates strategies that foster an intrinsic goal adoption and a self-endorsement of value in academic reading. By using the VTS, it is expected that students will develop an autonomous motivation for reading course materials and improve their engagement in the readings.

Fig. 2
figure 2

A screenshot of virtual tutee system

Full size image

The first design principle of the VTS refers to facilitating student identification with the role of tutor. When students identify themselves with a tutor role, they perceive the autonomy and competence that the role of teacher assumes. The primary means to encourage a role identification is to ensure that the role taker understands the expectations and responsibilities of the role. (Allen and Feldman 1976; Sarbin and Allen 1968). The VTS includes a guide video that walks students through the entire tutoring process and provides explicit directions on how to teach the virtual tutees. This introduction video helps students learn details of the tasks to be completed to fulfill the role of tutor.

The second principle is concerned with the provision of choices available to tutors in order to further support students’ feelings of autonomy. Providing a choice is one of the most effective ways to foster autonomy (Ryan and Deci 2000). The VTS offers three options. First, student tutors determine the tutoring goals they would like to focus on in their tutoring lessons. The tutoring goals restrict the content of tutoring, so it is critical to let student tutors decide on their tutoring goals in order to ensure their autonomy. When student tutors were not allowed to set their own goals, only minimal learning was observed (Rohrbeck et al. 2003). In addition, student tutors can choose the lesson delivery format in the VTS. For example, they can either write a summary of the assigned reading or explain its key concepts. Providing a review of the material to tutees is one of the activities that student tutors often perform in human peer tutoring, which enables the tutors to engage in deep learning (Roscoe and Chi 2008). By allowing students to determine the method for providing the review, their feelings of autonomy and commitment to tutoring can be further enhanced. Lastly, students choose the tutee they want to teach. They are given a list of six different avatar characters. Based on the distinct profile information of the characters (e.g., gender, hobby, hometown), students select one they would like to teach.

The third principle focuses on simulating social interactions between a student tutor and his or her virtual tutee to support the need for relatedness. In particular, three design strategies are implemented to augment the social presence of virtual tutees. One strategy is to have the virtual tutee ask questions. Answering a tutee’s questions is an essential element of tutoring that fosters student tutors’ deep learning (Roscoe and Chi 2008). As such, the VTS features the question-answering activity as the major task of the VTS. The virtual tutees ask a series of questions regarding the content of the readings, and the student tutors answer them. Another strategy is that the virtual tutees communicate positive attitudes toward learning by sending a message indicating their interest in the lesson. Such positive messages from the tutees can increase student tutors’ motivation by serving as positive feedback for tutoring as well as encouraging them to model the virtual tutees’ positive attitudes. Finally, the last strategy is to design the virtual tutoring to take place over a longer period of time. Research has found that students improved their performance over time with the repetitive use of a teachable agent (Schwartz et al. 2009). A prolonged interaction with the virtual tutees may also enable students to enhance their involvement in the role of tutor and develop their commitment as tutors.

Lastly, the VTS is designed to foster an intrinsic goal and perception of a meaningful value for reading, which is the fourth design principle of the VTS. Before students begin tutoring, the virtual tutees send a message that expresses their intrinsic aspirations for learning in the course and acknowledge the importance of reading course materials (see Fig. 3). This tutee message can serve as a prompt for students to model their virtual tutees’ positive motivation for reading.

Fig. 3
figure 3

A virtual tutee message conveying her perception of task value and task interest

Full size image

Based on the four design principles, the VTS prototype was developed and refined through two iterations of pilot testing (for details on revisions made in the VTS, see Park and Kim 2014). The goal of the study was to investigate the effectiveness of the VTS for enhancing students’ reading motivation, engagement, and performance. We were interested in examining the effects of the VTS after students had used it for a sufficient number of times. In the current study, the VTS was implemented throughout the entire semester (i.e., four times). Accordingly, the study addresses the following three research questions:

  1. 1.

    How does the VTS influence students’ motivation for completing course reading assignments?

  2. 2.

    What impact does the VTS have on students’ engagement (behavioral, cognitive, and emotional) in course reading?

  3. 3.

    What is the effect of the VTS on students’ reading performance?

Methods

Research design

This study incorporated a mixed methods design to bring together different sources of data on student engagement in reading and provide rich evidence-based accounts of the study findings (Creswell 2009). Engagement manifests in different forms and includes various indicators of engagement depending on how researchers operationalize it (Eccles and Wang 2012; Finn and Zimmer 2012). In this study, we collected both quantitative and qualitative data to adequately capture the dynamic nature of engagement in reading (Fredricks and McColskey 2012). Quantitative data were obtained via Likert-type surveys and qualitative data were collected through an open-ended survey and student interviews. Table 1 summarizes the alignment of research questions with data collection methods and analysis strategies. On each research question, the group who used the VTS was compared to the group who did not use the VTS (i.e., comparison group) following a quasi-experimental pre- and post-test design (Pedhazur and Schmelkin 1991).

Table 1 Alignment of research questions with data collection methods and analysis strategies
Full size table

Settings and participants

This study was conducted in an introductory educational technology course at a large public university in the southeastern United States. Participants were recruited from four sections of the course. Two different instructors taught the four sections. The first instructor, who had 9 years’ experience teaching this course, was in charge of three sections. The second instructor had 5 years’ experience of college teaching and had taught this course for a year prior to participating in this research. Individual sections were based on the same curriculum and lesson plans. The course was designed primarily to teach pre-service teachers about integrating technology in their classrooms; however, students in this course represented various majors including communication sciences, recreational studies, mass media, public relations, psychology, and education. The majority of the course work involved participating in a series of hands-on activities to learn how various technologies could be used for teaching and learning. Periodically, students were instructed to read assigned course materials, and the reading assignments formed 15 % of the course grade. The reading materials were compiled from various sources including a textbook chapter, magazine article, and blog posts.

Seventy-one students from the four course sections initially volunteered to participate in the study, but one student failed to complete the post surveys. Of the 70 participants, 21 failed to complete all four reading assignments. Only participants who completed more than two reading assignments (out of four) were included in the data analyses because the focus of the study was to examine the prolonged use of the VTS. Inclusion of these students yielded 63 participants in total. Fifty of the students were female (79.4 %). The majority (77.8 %) were Caucasian (n = 49) with 12.7 % African American (n = 8), 4.8 % Asian (n = 3), 3.2 % Hispanic (n = 2), and 1.6 % other (n = 1). Study participants were at various academic levels; eight freshmen (12.7 %), 19 sophomores (30.2 %), 17 juniors (27 %), and 19 seniors (30.2 %). The average participant age was 20.17 years (SD = 2.8).

Independent variables

The four sections of the course were randomly assigned to either the treatment or comparison group. Both groups received the same reading materials as take-home assignments. Students in the treatment group completed the virtual tutee system (VTS) while students in the comparison group completed the online reading guide (RG). The total number of participants in the VTS group was 31 (M = 5, F = 26), and the RG group included 32 (M = 8, F = 24) participants (see Fig. 4). The two groups included an equivalent number of students in each academic level. Also, the two groups were equivalent in terms of their GPA scores (p > .05).

Fig. 4
figure 4

Experimental group assignment of the study

Full size image

Treatment group

Students in the treatment group taught their virtual tutees about the content covered in the reading material for each assignment. When students signed up for the VTS, they selected their virtual tutee from a list of six options. The available virtual tutees were represented as human-like avatar characters, each of which had a different appearance. Each virtual tutee character was provided with a profile containing information that included hometown, major, year in college, and hobbies. Before beginning to teach the virtual tutee, students were directed to watch a guide video on how to navigate the VTS and teach the tutees so that they understood what they were expected to accomplish in tutoring sessions.

Once students entered a tutoring session, their first task was to determine tutoring goals. They were instructed to choose one or more goals they wanted to focus on in their tutoring sessions from a list of provided goal statements. Then, they were asked to provide a lecture note on the content they had read by either writing a summary or explaining the key concepts. No matter which type lecture note students chose, they were asked to address the same list of reading questions. After students completed the lecture note, their virtual tutees asked a series of questions expressing something they did not understand about the reading. Students responded to each question by typing an answer, which was the last part of the tutoring session. Tutee questions were provided through written text and were not vocalized. The questions used in the lecture note and the tutees’ questions were equivalent to questions used in the reading guide of the control group. They were reworded to appear as if the tutees were posing the questions (see Table 2). Some tutee questions were more elaborated by sharing the tutee’s perception about the content (see the third example in Table 2). Students repeated the same procedure for every tutoring session: determining tutoring goals, providing a lecture note, and answering tutee questions. The VTS in this study included four tutoring sessions that corresponded to the four reading assignments.

Table 2 Examples of modification of reading questions
Full size table

Comparison group

Students in the comparison group completed a Web-based reading guide for each reading assignment. The reading guide was composed of a series of four to five questions students were to answer. Participants were provided with a URL to each reading guide presented on a survey-creation website (see Fig. 5). To complete the reading guide, students typed their answers to each question in a specific field and clicked the submission button at the end of the reading guide.

Fig. 5
figure 5

Screenshot of a reading guide

Full size image

Data collection

Reading motivation

The Learning Self-Regulation Questionnaire (SRQ-L) (Black and Deci 2000) was used to measure the degree to which students demonstrated autonomous or controlled motivation for reading. Each item in the SRQ-L provides different reasons for why people engage in learning activities that represent either autonomous or controlled motivation. To match the context of the current study, the SRQ-L was slightly modified. The modified version of SRQ-L asked students to rate different reasons for completing the assigned readings on a seven-point Likert scale ranging from 1 (not at all true) to 7 (very true). An example item of the controlled motivation scale is, “I read the assigned readings because the instructor would have thought badly of me if I didn’t do the assignments”; an example item of the autonomous motivation is “I read the assigned readings because I felt like it was a good way to improve my understanding of the course material.” Reliability of the SRQ-L was acceptable in the current study with the alpha coefficients of .90 and .72 for autonomous motivation and controlled motivation, respectively.

Reading engagement

Reading engagement is defined as interacting with text (Guthrie and Wigfield 2000), and it can manifest in three different modes: behavioral, cognitive, and emotional (Fredricks et al. 2004). Behavioral reading engagement refers to observable behaviors that indicate students’ involvement in reading (Guthrie et al. 2012). In this study, we examined several indicators of behavioral reading engagement. First, we calculated a percentage of students who completed all four reading assignments in each of the two groups. Also, we developed three survey items to assess reading behaviors and time spent on reading. The first question asked students to choose the statement that best described their reading behavior among four options ranging from 1 (I mostly skimmed the text in order to get just the main ideas) to 4 (I read the entire text very thoroughly). The second question of the behavioral engagement survey asked about the average time students spent reading, and the last item asked them to report the average time they spent completing each of the reading assignments. Internal consistency was calculated with the last two items and the alpha coefficient was .73.

Cognitive reading engagement is generally understood as a psychological, mental investment in reading and related to meaningful processing of text (Guthrie et al. 2004). To assess cognitive reading engagement in this study, we measured students’ use of reading strategies while completing assigned readings by the Metacognitive Awareness of Reading Strategies Inventory (MARSI) (Mokhtari and Reichard 2002). The MARSI is composed of three subscales with a total of 30 items: global reading strategies, problem-solving strategies, and support reading strategies. The global reading strategies subscale consists of 13 items that relate to a global analysis of the text. An example item includes “I have a purpose in mind when I read.” The problem-solving strategies subscale contains eight items focusing on fix-up strategies in instances in which text has become difficult to read. An example includes “When text becomes difficult, I reread to increase my understanding.” The last subscale, support reading strategies, is composed of nine items referring to practical strategies for reading comprehension. An example item is “I underline or circle information in the text to help me remember it.” Students rated each item on a five point Likert scale ranging from 1 (I never or almost never do this) to 5 (I always or almost always do this). The alpha reliabilities in the current study were .81 (.77), .86 (.81), and .81 (.75) for global, problem-solving, and support reading strategies, respectively. Evidence for the validity of the scale was provided through a factor analysis in previous research (Mokhtari and Reichard 2002). Despite a good scale reliability and validity, several researchers commented on the limitation of using a self-report measure of reading strategies in that students do not accurately recall their use of the strategies (Cromley and Azevedo 2006; Veenman 2005). Nevertheless, we insisted on using this retrospective measure because it allows researchers to examine students’ reading strategy use without disrupting the natural flow of reading that can interrupt students’ motivation to read.

Emotional reading engagement is concerned with students’ affective reactions toward reading materials (Finn and Zimmer 2012). In this study, we used part of the Achievement Emotions Questionnaire-Mathematics (AEQ-M) (Pekrun et al. 2005) to measure students’ emotional engagement in the readings. The AEQ-M is a multidimensional self-report survey that assesses students’ achievement emotions experienced with mathematics in three different situations: attending class (class-related), studying and doing homework (learning-related), and taking tests and exams (test-related). For the purpose of the current study, the learning-related part of the AEQ-M was modified to assess students’ emotional experiences, specifically boredom, enjoyment, and anger, with reading assignments. In the original AEQ-M, an example item of boredom is “Just thinking of my math homework assignments makes me feel bored.” In the current study, this item was modified to “Just thinking of reading assignments in this class makes me feel bored.” The items of enjoyment and anger were reworded in a similar way. Each subscale of boredom, enjoyment, and anger contained three items, making nine items in total. Students responded to each item on a five-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The Cronbach’s alphas of the three subscales in the current study were .86 (.77), .75 (.67), and .84 (.78) for boredom, enjoyment, and anger, respectively.

In addition, an open-ended survey and a semi-structured interview protocol were developed to explore reading engagement that may not have been measured by the Likert-scale surveys. Past research indicated a limitation of reading strategy instruments in that students are not proficient at accurately recalling their previous use of the strategies (Veenman 2005). We expected that open-ended questions could prompt students to provide a detailed description of the behaviors, emotions, and thinking processes that they experienced while completing the reading assignments.

The open-ended survey contained five questions asking students to describe different aspects of the reading assignment. The sample questions were, “How did you like completing reading assignments in this course?” and “Do you think the VTS (or reading guides) influenced your understanding of the reading materials?” A semi-structured interview protocol included questions similar to the ones used in the open-ended survey. The protocol was prepared to gain a more in-depth understanding of students’ engagement in course readings. The protocol was used primarily as a guide, and the interviewer was allowed to change and add questions (Roulston 2010). All the interviews were audio-recorded and transcribed verbatim.

One of the researchers individually open-coded student responses in the open-ended survey and interviews through multiple readings and classified the codes into categories of reading motivation and engagement (e.g., behavioral engagement, cognitive engagement, emotional engagement) (Crabtree and Miller 1992). We specifically used an open-coding strategy because reading engagement can take various forms and its indicators cannot be confined (Strauss and Corbin 1990). The other researcher evaluated the codes and categories. Together the researchers refined the codes and reached consensus on the code categories. Additional categories were developed for the codes that did not fit into the existing categories. Through an iterative process, the researchers grouped codes and categories into common themes (Fereday and Muir-Cochrane 2006). The final themes were compared with the survey results to examine if the two data sources yielded consistent findings.

Reading performance

Student reading performance scores were obtained through an evaluation of students’ responses provided in the reading assignments. Each reading assignment contained four to five open-ended comprehension questions related to the content of the reading. These queries were composed of different types of reading comprehension questions such as reorganization, inference, evaluation, and personal response (Day and Park 2005). In order to evaluate students’ answers to these reading questions, two researchers developed rubrics in consultation with the course instructors. The rubrics were concerned with whether students provided accurate and adequate answers to the questions (scale: 0–5 points). For example, on the question, “What is the teacher’s role in technology-supported communication projects?,” students received 5 points if they provided thorough correct answers based on the reading, 3 points if their answers missed out some information, and 0 point for incorrect or irrelevant answers.

Student scores on the four reading assignments were summed to calculate a total reading performance score. The maximum performance score was 90. To ensure validity of the performance scores, two researchers independently reviewed and rated students’ answers on the reading assignments. Neither rater had access to the group assignment information while evaluating the data (i.e., blind data). Interrater reliability for the raters in the first independent evaluation was found to be an intra-class correlation coefficient (ICC) = .83. The two raters then discussed and refined the scoring criteria on the rubric and independently reevaluated the items. After the second independent evaluation, the interrater reliability increased to ICC = .97. The two raters further discussed and reviewed each item with dissimilar scores until they reached agreement.

Teacher autonomy support

According to self-determination theory, the teacher’s motivating style is one of the critical factors affecting student motivation and engagement (Reeve 2012). Thus, students’ perceived support for autonomy from their instructors was considered an important covariate of the study. In this study, the Learning Climate Questionnaire (LCQ) (Williams and Deci 1996) was used to assess possible differences in the two instructors’ autonomy support and understand their effect on student engagement in reading. The LCQ is a self-report survey that assesses the degree to which the instructor supports student autonomy. We used an abbreviated version of the LCQ containing six items. Example items include, “My instructor listens to how I would like to do things” and “I feel that my instructor provides me choices and options.” Students rated each item on a seven-point Likert scale ranging from 1 (strongly disagree) to 7 (strongly agree). The current study found a high internal consistency of the LCQ with an alpha coefficient of .93.

Procedures

Participants for the treatment and comparison groups were recruited at the beginning of the semester. Those who volunteered to participate completed a demographic survey, the MARSI, and the AEQ as pre-intervention surveys on the day they were recruited. During the third week of the semester, they were given their first reading assignments. To complete the assignment, students in the RG group answered the reading guide questions online whereas students in the VTS group taught their virtual tutees via the VTS. Participants completed three more reading assignments in the same manner throughout the semester. Students received the second reading assignment 2 weeks after the first assignment; the third reading was assigned 6 weeks after the second; the fourth reading assignment was completed 2 weeks after the third assignment.

Once students had finished the fourth reading assignment, they were asked to fill out the behavioral engagement survey, MARSI, AEQ, SRQ-L, LCQ, and open-ended questionnaire as post-intervention surveys. Also, students were recruited for participation in follow-up individual interviews. A total of 11 students participated in the individual interviews: six from the VTS group and five from the RG group. Those who agreed to participate in the interviews received an incentive of an extended due date for their course assignment. All the interviews were conducted face-to-face in a small-sized conference room, with each interview lasting approximately 30 min.

Results

Quantitative data results

We first calculated a reading completion rate for each group, one of the measures of behavioral engagement. Twelve students in the RG group (33.3 %) did not complete at least one reading assignment out of four as compared to nine in the VTS group (26.4 %). The number of students who missed more than one assignment (two or more) was four for the RG group (11.1 %) and three for the VTS group (8.8 %). As the focus of the study was to investigate changes in reading engagement as a result of using the VTS, these seven students were eliminated in the subsequent data analyses examining students’ engagement in reading. The descriptive statistics for the study variables are summarized in Table 3.

Table 3 Mean pre-survey scores and mean and adjusted mean post-survey scores for dependent variables
Full size table

To compare the levels of students’ autonomous and controlled motivation for reading between the two groups, we performed a multivariate analysis of covariance (MANCOVA) with perceived autonomy support (instructor influence) as a covariate. GPA was not included as a covariate because it did not yield a significant relationship with any of the motivation scores. We did not find a significant difference in this analysis, Λ = .95, F(2, 59) = 1.48, p > .05. Both groups demonstrated a moderate degree of autonomous (M RG  = 4.82, M VTS  = 4.83) and controlled motivation (M RG  = 5.21, M VTS  = 5.70) for reading.

Reading behavior and assignment completion time, the indicators of behavioral reading engagement, were compared between the two groups with a multivariate analysis of variance (MANOVA). Time spent on reading was not included in the analyses due to a violation of the assumptions. The average assignment completion times between the two groups were not significantly different, Λ = .98, F(2, 60) = .40, p > .05. The means of the reading behavior were 2.42 and 2.53 for the VTS group and the RG group respectively, indicating that both groups were engaged in some degree of thorough reading (see Table 3).

In order to compare the three subscale scores of the MARSI between the two groups, a MANCOVA was conducted with covariates of the pre-survey scores and GPA. GPA was entered as a covariate because the use of reading strategies has been associated with academic achievement in the literature (Pintrich and De Groot 1990; Pintrich and Zusho 2007). Perceived autonomy support was not included as a covariate without a significant relationship with any of the MARSI scores. The results indicated that two groups demonstrated a similar pattern of reading strategy use, Λ = .87, F(3, 55) = 2.38, p > .05.

Using the pre-survey scores on AEQ and perceived autonomy support as covariates, the MANCOVA yielded a statistically significant difference on the scores of the three emotion scales between the two groups, Λ = .59, F(3, 55) = 12.33, p < .001, partial \(\eta_{p}^{2 }\) = .40. To further examine the significant effect on the multivariate analysis, follow-up univariate ANCOVAs were conducted on each of the three emotions with a Bonferroni correction. According to the ANCOVA results, the two groups were significantly different in enjoyment, F(1, 57) = 23.67, p < .001, partial \(\eta_{p}^{2 }\) = .29. No significant difference was found in two other emotions (anger and boredom).

Lastly, after controlling for students’ GPAs, a significant difference was found on students’ performance scores, F(1, 59) = 7.09, p = .01, partial \(\eta_{p}^{2 }\) = .10. The adjusted mean scores of reading performance were 72.09 for the VTS group and 63.80 for the RG group.

Qualitative data results

Data from the open-ended survey and the interviews indicated that the VTS (n = 7) or the reading guide (n = 3) encouraged their behavioral engagement in reading. Students reported that the VTS or the reading guide made them complete the assignments. For example, a student from the VTS group said, “It [VTS] forced me to sit down and learn even on days when I wasn’t feeling particularly motivated.” A student in the RG group similarly commented, “I probably wouldn’t have read the readings at all if it were not for the reading guide questions.” Additionally, four students in the VTS group mentioned how the VTS fostered thorough reading over skimming. For example, a student said, “It just makes you go back and re-read for content and not just to get it over with.”

Also, students hinted that the reading guides (n = 14) or the VTS (n = 20) provided cognitive benefits. Students reported that the reading guides or the VTS helped identify the main ideas and enhanced comprehension of the texts. For example, a student from the RG group reported, “They [reading guides] helped me focus on which parts of the reading I needed to gather information most from.” A student in the VTS group commented, “The Virtual Tutoring Lab helped me reinforce my understanding of what I’d read in asking me to elaborate on certain points and summarize certain ideas.” These reports from the students implied that comprehension questions prompted them to organize what they had learned from the readings and reflect on it, which led to a better understanding of the readings. Indeed, a student in the VTS group reported that the tutee questions made her “engage in critical thinking.”

Several students in the VTS group (n = 4) further specified that tutoring activities promoted their cognitive engagement in reading. A student reported in the open-ended survey, “By having to teach someone else the material, you had to understand it yourself first, so that was good motivation.” Another student wrote, “The fact that I had to turn the subject around and teach it to someone else really helped me (understand the assigned materials).” The following interview excerpt also indicates the benefits of being a tutor:

The virtual tutoring was pretty interesting. It gives you a chance to kind of be a teacher I guess you’d say … and so in doing that I was able to reinforce the topics for myself as well so I think it was beneficial.

To the question of how well students liked reading assignments, both student groups reported some degree of enjoyment though the reasons were slightly distinct between the groups. Most students in the RG group expressed pleasure with the brevity and easiness of the assignments. On the other hand, the VTS group found the process of completing the assignments interesting. Five students referred to the VTS as a novel and enjoyable way of learning and completing reading assignments (e.g., “I think using the virtual tutoring lab is a fun way of making students do the assignments”). Two of them commented in particular that answering tutee questions was a motivation to learn and read (e.g., “I liked reading them because I was motivated to answer the questions, which helped me understand key details”). Two other students reported enjoyment with completing the assignments online. A few students also (n = 3) showed a liking for having some choice in the VTS. For example, one student stated, “I thought it was interesting that we could choose who we wanted to tutor. I also liked that we were given choices in what we wanted to discuss in our reading assignments.” The interview excerpts below are representative remarks that students made about the appeal of the VTS.

Yeah. I think it was interesting, like pick a person. I remember reading the descriptions of each person and picking the one that I think I most liked or something. Yeah, so that was interesting, it was different. And then when they’d ask me questions… that was a lot better than just like filling out a worksheet or something like that.

I liked having a little person talk to me. I thought that was cool…. I liked how you could choose like summary or key concepts because it personalized it.

It feels somewhat interactive because you get to choose the person you tutor and the way they present the questions is like they are speaking to you in a conversation.

Nevertheless, some students reported discomfort with the VTS. Six students specifically reported that they did not enjoy the reading assignments although they acknowledged that the VTS was helpful. Several students using the VTS complained about spending too much of their time answering the questions. They reported that the VTS required them to do more work than they would have had to do for reading assignments in general. Some students mentioned that the type of questions were less reflective of questions that tutees would ask, which created confusion. One interviewee said, “I guess the questions that were asked … didn’t seem like a real person would ask.” Another student attributed the extended time she spent in completing the assignments to the obligation she felt to answer the tutee questions:

I think it would help like that and maybe if the question was more specific so that I wouldn’t have to write so much because I felt like some of them had very open-ended, broad answers and I didn’t feel comfortable just talking a little bit so I’d write a lot and then. I think at first the concept of having to teach made me feel like I had to talk about it as much as I can…. I felt like I had to cover all the aspects of the material, like especially if I chose the summarizing one, then I would summarize it. But then I’d also take the time to expand on each point that I wrote about. I just wasn’t sure how much would be okay to write or enough.

Lastly, technical difficulties with the VTS were reported. When answering the tutees’ questions, students had to proceed to the next page on which a blank field was presented to type the answers (see Fig. 6). However, once students moved to the answering page, the tutee’s question was no longer available for viewing and they had to leave the page to review the question. Many students (n = 7) reported this issue to be an inconvenience. For example, one student said, “I wish the questions would not disappear when I clicked next to answer the question. I had to open a document, paste the question, write my answer, and copy the answer, then paste the answer in the answer box.”

Fig. 6
figure 6

Screenshot of providing answers to a tutee question in the VTS

Full size image

Discussion

The purpose of the study was to examine the effectiveness of the VTS for enhancing college students’ engagement in academic reading. We compared students who used the VTS to those using an online reading guide by examining their reading motivation, engagement, and performance. Based on the theoretical foundation of and previous work on the VTS, we expected that students who used the VTS would be more likely to show greater reading motivation, engagement, and performance. The study findings yielded partial support for the effectiveness of the VTS.

Both reading guides and the VTS seem to have stimulated students’ inclination to read. Both groups reported that they would have been less likely to read without the assigned activities. Having to produce some kind of output after reading (i.e., answering reading questions), students were more or less forced to complete the assignments. Indeed, students reported a higher controlled motivation than an autonomous motivation toward the readings. In addition, the findings from the Likert-type scales indicated that both groups were engaged to a similar degree in thorough reading and reading strategy use.

A few findings indicated a deeper reading engagement among VTS students compared to students who completed the reading guides. First, more students in the VTS group (73.6 %) completed all four reading assignments as compared to those in the RG group (66.7 %). Also, more students in the VTS group tended to acknowledge how the VTS had promoted their deep reading and thinking. In particular, several students commented on the effects of tutoring on their reading behaviors. Moreover, students in the VTS group performed significantly better on the reading assignments than did those in the reading guide group. In other words, students in the VTS group provided higher quality, more accurate answers on the reading assignments than did those in the RG group. Such higher performance in the VTS group implies that students in the VTS group engaged in a deeper level of information processing (“deep learning”) as compared to students answering the reading guide questions.

Reports on the open-ended survey and interviews hinted that the act of teaching encouraged students to elaborate on their answers and fostered a greater cognitive effort on the question-answering activities. More students in the VTS complained about the extended time they spent completing the assignments although the two groups reported an equivalent amount of assignment completion time on the behavioral engagement survey. This implies that students in the VTS group may have expended greater mental effort than those in the reading guide group. This finding is consistent with prior research that student tutors provided elaborative and reflective responses rather than simply conveying information when answering tutees’ questions (Roscoe and Chi 2008).

According to role theory, serving as a tutor assumes taking on the responsibility for a tutee’s learning and facilitates a commitment to learning (Robinson et al. 2005). Students could have provided answers merely sufficient for what the questions asked. Because students identified themselves with the role of tutor in the VTS environment, students instead exerted a greater cognitive effort and engaged in a deeper level of thinking to successfully teach their tutees. Indeed, a couple of students reported that answering tutee questions motivated them to read and learn from the readings. This finding suggests that the question-answering feature in the VTS facilitates students’ identification with the role of tutor and could be considered as a key element in simulating a tutoring environment.

Nevertheless, the study found no significant difference in the use of reading strategies between the two groups. In the previous pilot studies, students improved their use of reading strategies after using the VTS (Park and Kim 2014). The absence of significant differences in students’ use of reading strategies in the current study may be attributable to the low level of difficulty of the texts that students read. The readings in this course were more akin to informal reading materials such as a magazine article, a blog post, and an overview of a research summary. Students spent about 10–40 min reading each of the readings. Unlike traditional textbooks, these readings were relatively short and written with simple, plain language and vocabulary. Thus, it is possible that students did not need to use sophisticated reading strategies to understand the study’s reading materials. Additionally, it is also possible that students failed to recall their reading strategy use. Previous research has indicated that a self-report questionnaire might not accurately measure students’ actual use of reading strategies because it relies on their retrospection (Veenman 2005).

With regard to emotional engagement in reading, inconsistent findings were reported. In the open-ended survey, the majority of students in the VTS group reported pleasure in using the VTS. Students seemed to like the tutoring activities of the VTS, and some in particular enjoyed the act of tutoring itself. The interview data indicated similar findings. However, when the two groups were compared with regard to survey scores on their emotional experiences with the reading assignments, students in the RG group indicated a higher enjoyment of the readings than did those in the VTS group.

Inconsistency in students’ reports on their emotional experiences may be understood as another evidence for greater cognitive exertion among students in the VTS group. According to the general definition of emotional engagement (Fredricks et al. 2004), the low enjoyment found in the VTS group suggests a low emotional engagement in reading. However, such displeasure with the reading assignments can be experienced as a result of deep cognitive processing. A seminal study by Zeidner (1987) found that people tend to experience negative emotions when working on a task with high demand for a cognitive process (e.g., working memory, information retrieval). In fact, some students in the VTS group expressed discomfort with spending more time on the reading assignments than they had expected. That is, the more intense cognitive load among students in the VTS group (i.e., deep processing) may have contributed to the perception of greater time consumption, which may also have caused low enjoyment in reading. Thus, the low enjoyment reported by the VTS group may a by-product of deep cognitive engagement in reading.

Lastly, the two groups did not differ in their motivation for completing the assigned readings. It was expected that students in the VTS group would show a higher autonomous motivation than students in the RG group because the VTS was designed to embrace the sources of autonomous motivation. However, both groups demonstrated a higher controlled motivation than an autonomous motivation. This negative finding could be explained by referring back to SDT. According to the theory, the self-endorsed value of a task is one of the critical elements for students to develop autonomous motivation (Deci and Ryan 2000; Vansteenkiste et al. 2004). Thus, students need to appreciate the value of completing the reading assignments to demonstrate autonomous motivation. However, several students mentioned during the interviews that the reading topics were not of interest to them because they were not related to their majors. In addition, the importance of the reading assignments in this course was relatively low compared to other college courses because it was not a typical lecture-based course and did not involve a formal examination. The reading materials did not necessarily introduce technical vocabularies or complex concepts that students needed to learn. Together, these circumstances may have lowered the perceived value of the readings in this particular course. In light of this, it is possible that students in the current study may not have been able to fully capitalize on what the VTS could offer, lessening an autonomous motivation for reading with use of the VTS.

In conclusion, the study findings demonstrate the potential of the VTS as a learning tool that promotes students’ deep-level thinking. Although students in the VTS group did not show a greater autonomous motivation for reading nor a greater use of reading strategies, they still achieved a higher performance on their reading assignments than did those in the RG group. The study findings imply that the VTS facilitated students’ deep cognitive processing of a text when completing the reading assignments. In particular, serving as a tutor seemed to encourage students to take the assignments more seriously and exert greater cognitive effort.

Limitations of the study

One limitation of the study lies in the use of a self-report survey to measure engagement. To assess behavioral engagement in reading, students were asked to think back and recall the average time they had spent reading. However, it is possible that students might not have remembered the exact amount of time they had spent in reading. Although students were asked to report both the time they spent reading and the time they spent completing each assignment, some did not differentiate between these two. The use of self-report measures of reading strategies raised the same issue (see Cromley and Azevedo 2006). Thus, the survey data in the study could have underestimated or overestimated the true level of reading engagement.

In addition, students’ reading skills were not considered in this study. Students’ past performance and competency levels have a significant impact on their subsequent motivation and engagement (Atkinson 1957; Weiner 1985). It is possible that students’ reading skills might have moderated the effects of the VTS on reading motivation and engagement in addition to influencing students’ reading performance. Although their GPAs were used as a covariate to control for their prior skills, a measure of reading skills would have been more precise since the study focused on reading engagement and performance.

Another limitation is related to the recruitment of interview participants. In the current study, interview participants were recruited toward the end of the semester. In order to encourage participation, they were given the incentive of an extended due date for their course assignment. Thus, it is possible that students with a lower performance in the course were more likely to participate in the interviews, which might have yielded the biased data. In future studies, purposeful sample interviews are recommended instead. For example, researchers may decide to select those who clearly demonstrate either a deep or shallow level of engagement in the open-ended survey.

Finally, students encountered technical problems with the VTS. Several students reported that they had difficulty logging in because they had forgotten their password, but the password resetting function in the VTS did not work properly. Also, on two occasions, the VTS failed to save students’ answers. In future studies, these technical issues should be resolved so that students do not experience any discomfort using the tool.

Suggestions for future research

There is a need for further research to better understand how the VTS influences students’ deep reading. Although the study found partial evidence that students engaged in a deeper level of thinking by using the VTS, the underlying cognitive processes are not still known. Moreover, we did not find a significant difference in reading strategy use between the VTS and the reading guide groups. Future research may consider examining the relationships between different levels of reading strategies and the VTS. Also, other ways to measure deep reading are recommended. For example, future researchers may use eye-movement data to assess students’ cognitive processing while reading (Rayner 2009; Reichle et al. 2010).

Another area considered for future research is the type of questions asked by virtual tutees. The ones used in the study were generated based on the reading guide questions that the course instructor had been using in the course. To promote students’ deep thinking, the instructor had created higher-order questions. Many of these questions involved asking for personal judgments, beliefs, and opinions as well as discussion-type questions that could have more than one answer (e.g., Among these four dimensions of creativity, with which dimension do you struggle the most?). Several students indicated that these questions were not reflective of authentic tutee questions. Although the questions were adapted to be suitable for the tutoring environment, they were still more akin to teacher questions rather than questions from a tutee seeking knowledge (Graesser and Person 1994). In such case, students’ identification with the role could be interrupted, and their feelings of autonomy be diminished. Future research needs to identify the types of tutee questions that foster an authentic tutor-tutee relationship and enhance students’ commitment to the tutor role.

The last implication of the current study is related to the measure of emotional engagement in reading. In many research studies, positive emotions such as enjoyment and pride are regarded as deep emotional engagement in contrast to emotional disaffection (e.g., Skinner et al. 2008). However, the study findings imply that deep engagement may not always result in positive emotions. Although students in the VTS groups seemed to exert a greater cognitive effort, they reported lower enjoyment than did students in the RG group. As noted above, when students engage in tasks with high cognitive demands, they tend to experience negative emotions temporarily until the cognitive discrepancies are resolved (Zeidner 1987). Thus, the measure of emotional engagement based solely on valence (positive versus negative) might not be applicable in some contexts, in particular, a circumstance in which students are involved in complex cognitive tasks (Pekrun 2006). Future researchers may consider using multiple measures to accurately assess students’ emotional engagement in learning, as suggested by Fredricks and McColskey (2012).