The broad aim of this and related research is to identify conditions that support vicarious learning processes in multimedia environments and how these conditions can assist both distance learning and standard classroom instruction (Anderson et al. 1995; Chi et al. 2008; Cox et al. 1999; Craig et al. 2002; Gholson and Craig 2006; McKendree et al. 1998; McNamara et al. 2004). In these vicarious multimedia environments, learners are not the addresses of the educational material they are attempting to master. Instead, they can engage in only cognitive activity, neither controlling the material’s source nor interacting with it in any way (Craig et al. 2006; Mayer 2001, 2002; Rummel and Spada 2005). Thus, the only activities available to learners in these vicarious environments are cognitive. The learner must actively process incoming information into coherent representations that can be integrated into existing mental models and schemas (Gholson and Craig 2006; Mayer 1997, 2001; Sweller 1988, 1999). In the next section we briefly discuss some earlier research that explored the role of deep-level reasoning questions in facilitating these cognitive activities during vicarious learning.

Deep-level reasoning questions and vicarious learning

Deep-level reasoning questions (hereafter deep questions) usually request one of the following three kinds of reasoning: the first is logical reasoning, which states the premise and conclusions in a syllogistic sequence. The second is causal reasoning, which articulates the states the events in a causal chain. The third is goal-orientation reasoning, which expresses the planning and goals of agents (Bloom 1956; Flavell 1963; Graesser and Person 1994; Piaget 1952, 1968). These questions ask the hearer to answer such questions as “What happens when…?”, “How does the…?”, and “Why was the…?”. The cognitive activities required by generating and answering such deep questions are involved in comprehension (Collins et al. 1980; Rosenshine et al. 1996) and problem solving (Graesser et al. 1996; Sternberg 1987).

Craig et al. (2000) used vicarious learning procedures to show that vicarious learners presented with such deep questions embedded in course content outperform similar vicarious learners presented the same course content without such questions. This deep questions effect was soon replicated using a variety of experimental and control conditions (e.g., Craig et al. 2004, 2002; Driscoll et al. 2003).

This deep questions effect during vicarious learning was next contrasted with learning gains obtained by learners who interacted with an intelligent tutoring system (ITS) called AutoTutor. This ITS, which holds a conversation in natural language with the learner, produces learning gains of 0.6–2.1 standard deviation units when compared to various control conditions (Graesser et al. 2004, 2001; VanLehn et al. 2007). AutoTutor’s script begins each topic with an overview followed by a difficult question for the tutee to answer. Sentences in an ideal answer to this question are decomposed into sentences containing a set of key concepts called expectations. Latent semantic analysis (Graesser et al. 2001; Landauer and Dumais 1997) and other semantic evaluation algorithms are used to assess the learner’s progress on each topic by comparing learner contributions to the content of the expectations. The dialogue moves of AutoTutor are subsequently sensitive to content that is covered in these semantic matches so that content gaps are eventually filled by the student or tutor. Thus, the dialogue moves of AutoTutor are dynamic and adaptive.

In two experiments Craig et al. (2006) contrasted interactive learning gains of students tutored by AutoTutor with gains of vicarious learners. In Exp. 1, Craig et al. (2006) compared pretest-to-posttest gains obtained by learners in interactive tutoring sessions with gains obtained in four vicarious conditions. One vicarious condition included dialogue containing deep questions. These questions preceded each content sentence in AutoTutor’s ideal answer and each expectation in the script. The other three vicarious conditions included dialogue with questions before some of the content sentences, monologue presentations of the content sentences, and videos of interactive tutoring sessions. The vicarious condition which included deep questions before each content sentence significantly outperformed the interactive condition as well as the other three vicarious conditions. In Exp. 2, Craig et al. (2006) showed the deep questions effect held when questions were embedded in monologue as well as dialogue.

Present research

Two considerations that limit the generality of the findings described in the last section are addressed in this new research. First, our earlier studies included only college students, but the new research involves eighth-to-eleventh grade students. Second, our earlier work investigated vicarious learning processes only in the domain of computer literacy, but the new research includes both the domains of computer literacy and Newtonian physics.

The present research contrasted the performance of eighth, ninth, tenth, and eleventh grade students who were tutored by AutoTutor with similar students in vicarious learning conditions. The eighth and tenth graders were asked to learn computer literacy, while the ninth and eleventh graders studied Newtonian physics. Three experimental conditions at each grade level included interactive tutoring sessions with AutoTutor, a vicarious-learning condition involving monologue presentations of each sentence in the ideal answer and each expectation (see previous section), and a vicarious learning condition which included deep questions dialogue in which the content of each sentence in the ideal answer and each expectation was preceded by a deep question. Based upon prior research, we predicted that vicarious learners presented with deep questions would outperform those in the monologue condition and perhaps perform as well as (learn as much) if not better than those in the interactive tutoring sessions with AutoTutor.

Method

Participants

This study included 342 participants: 74 eighth graders, 91 ninth graders, 90 tenth graders, and 87 eleventh graders. They were drawn from a public school in a mid south city. All students were tested in group settings of 11 or more in the classroom or in a laboratory at the University. Parental permission was obtained for each student prior to participation in the research.

Design, materials, and procedures

There were three experimental conditions at each of the four grade levels. The interactive condition involved participants interacting directly with an intelligent tutoring system by carrying on a dialogue with AutoTutor (112 participants). The monologue vicarious condition involved a virtual tutor asking one broad question at the outset of each topic and following this question with the sentences included in the ideal answer and expectations on that topic (116 participants). The dialogue condition involved a virtual agent asking a deep question presented by a second voice engine before each content sentence in the ideal answer and each sentence in the expectations (114 participants).

All participants at each grade level were randomly assigned to one of the three experimental conditions at the outset of their first session. After informed consent they were administered the Gates-McGinite reading test (MacGinitie et al. 2000), followed by a pretest. A 26 (three or four foil) multiple-choice questions (chance = 7 correct) was presented to participants asked to learn Newtonian physics. The test was developed by VanLehn et al. (2007) and was similar to the Force Concepts Inventory (Hestenes et al. 1992). A second version this test, used to evaluate learning gains, was administered at the completion of the study. On computer literacy two 24-item four-choice multiple choice tests (chance = 6) were used to evaluate pretest to posttest gains (see Craig et al. 2004, 2006). The two versions of the tests in each domain (both computer literacy and Newtonian physics) were shown to be equivalent in previous research (see Craig et al. 2004, 2006; Graesser et al. 2004; VanLehn et al. 2007). The two versions of the tests used in each domain were administered in counterbalanced order to learners in each condition. After completing the pretest they were presented with one of the three computerized conditions. Each computerized session was individually presented via a laptop computer located on the student’s desk. All tutoring sessions were approximately 37 min in length. Altogether, the complete study required most of each of three classroom sessions for each participant who was tested in the schools (a total of about 130 min) and about 130 min for those tested in the laboratory setting.

Results and conclusions

There were no significant differences on the Gates-McGinite test among the three experimental groups within any of the four grade levels. There were also no significant differences between computer literacy and Newtonian physics on the pretests, F(1,339) = 1.10, p = 0.90. There were no significant differences at pretest among the three experimental groups (interactive, monologue, and dialogue) at any of the four grade levels (largest F = 0.83, p = 0.42). There were also no significant effects of grade level on the pretest (see Table 1).

Table 1 Means, standard deviations, adjusted posttests, and pre-post Cohen’s d effect size for pretest and posttest data, by grade level and condition
Full size table

An analysis of covariance was performed on posttest scores in a 3 (experimental condition: dialogue versus interactive versus monologue) × 4 (grade: 8 vs. 9 vs. 10 vs. 11), using pretest scores as covariates. There were significant effects of experimental condition, F(2,329) = 6.35, p < 0.05, and grade, F(3,329) = 43.94, p < 0.001. The interaction did not approach significance (p > 0.80). The effect of grade reflected significantly higher scores among both ninth and eleventh graders than among both eighth (p < 0.01) and tenth graders (p < 0.01). Neither eight and tenth graders nor ninth and eleventh graders differed significantly from each other (p > 0.06 in each case). It will be recalled that eight and tenth graders learned computer literacy, while ninth and eleventh graders learned physics. A comparison of computer literacy with physics was significant, T (340) = 3.34, p < 0.001.

The result of most interest in the current study was the effect of experimental condition (see Table 1), which was significant, as indicated in the previous paragraph. Students in the dialogue condition, with a learning gain from pretest to posttest of 31%, learned significantly more than those in both the interactive (p < 0.04), with 14% gains and the monologue conditions (p < 0.05), with gains of 17%. The interactive and monologue conditions did not differ significantly from each other. These data indicate that the deep questions effect during vicarious learning is more robust than was previously demonstrated, in that it had been obtained only among college students in the domain of computer literacy (see Craig et al. 2006; Gholson and Craig 2006).

Before concluding, we highlight some of the limitations in our current knowledge and some issues that may need to be addressed by those concerned with distance learning, and computer-based courses in general (Anderson et al. 1995; Graesser et al. 2001; McNamara et al. 2004). First, to date, our research on the role of deep questions during vicarious learning has involved only the domains of computer literacy and Newtonian Physics. In what other domains does it hold? Second, are the deep question categories (i.e., causal antecedent, causal consequent, comparison, enablement, instrumental procedural, interpretation) that were used (see Graesser and Person 1994, pp. 110–111) equal in supporting knowledge construction during vicarious learning? Future research should answer this question, identify the features of these categories that support construction, and determine what other features of discourse support these processes.

In conclusion, we note that the deep questions effect is robust in that it generalizes (a) from college students to middle and high school students, and (b) from the relatively declarative domain of computer literacy to the more causal domain of physics. While, as indicated directly above, much further research is needed, the current findings do have some practical implications. First, embedding deep questions into course content presented via computers used in the classroom or in distance learning environments may produce real benefits for learner at little cost, in terms of time or money. Second, the finding that deep questions embedded in course content can perform at least as well (or outperform) an ITS, has clear implications for return on investment, in that it is much more costly to produce an ITS than it is to simply embed deep questions in a computerized script.