Abstract
This paper presents the results of student engagement with eBRAVO, an Intelligent Tutoring System designed to support students’ development of reading comprehension strategies. The eBRAVO curriculum is a personalized experience based on the students’ previous engagement with the tool as well as their demonstration of deep comprehension of the current materials. This personalization may include support in the form of video lessons that target the comprehension strategy with which the reader has recently struggled embedded within the chapter context the reader was currently working. This paper outlines the results from a deployment during a summer program supporting students reading ecology content, and shows that students are clearly distinguishable into categories that denote their patterns of engagement with these videos. It also discusses how these results connect to comprehension assessment results within the system and at a unit level and the implications these results have for the design of future classroom intervention systems.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
Science education is changing. Increasing emphasis is being placed on students in science-as-practice [10] that is helping students understand science not only as a body of knowledge but also as a set of practices for developing and warranting knowledge claims about the natural world. One of the most important aspects of this process is reading scientific texts. The recent report A Framework for K-12 Science Education [10] cites “Obtaining, Evaluating, and Communicating Information” (pg 74) as one of the key science and engineering practices. This practice requires students not only to interpret text, but also to compare and evaluate texts in the context of explaining phenomena and solving problems. By twelfth grade, students are expected to be able to read and critique published scientific texts.
In this study, we present the analysis of student engagement with eBRAVO, an ITS designed to support students’ reading comprehension in strategies by leveraging the Construction-Integration (CI) model of reading comprehension [21, 22]. We use a learning analytics [29] approach to understand student activity within eBRAVO using a feature driven sequential analysis approach [27]. We explore the distribution of engagement variables to understand different patterns of use and how they map to in-system measures of student comprehension and unit-level measures of student reading comprehension abilities.
2 Background
2.1 Reading Comprehension and Science Education
Reading to learn [6] is a critical skill to be supported in late primary and secondary education. This represents a significant challenge, since at present many students do not reach basic proficiency in reading comprehension for their grade level, including one third of fourth grade students and one quarter of eighth grade students [9]. Additionally, while great national and international effort has gone into supporting students’ reading skills, a significant gap exists in supporting the reading of domain-specific text, especially in science. Reading scientific texts requires skills above and beyond general reading comprehension [13], in part due to the complex nature of the writing style and the importance of synthesizing content across multiple texts, figures, charts, and other sources of information [10]. To be successful in school, students need to be able to read content-area texts well enough to learn and remember new content, and they must be able to apply what they know in future reading and learning situations, as well as in novel contexts. When students begin upper elementary school, they are expected to read content-area texts (often in the form of a textbook); however, they are rarely provided with instruction on how to read informational and expository texts.
Compounding the problem, educators are often ill-equipped to address these domain-specific reading issues. Science teachers are not language arts experts. Their training and expertise focuses on scientific concepts for students to learn to independently examine scientific information. This is especially problematic when students read expository texts, not only because of the unfamiliar subject matter, but also due to the higher density of ideas, lack of coherence, and more complicated reference and structure employed in such texts [23]. There is a sudden jump in the complexity of content area readings in middle and high school, such that many of the popular textbooks are in fact as complex as college-level material in their syntax, vocabulary, and presentation as well as in content [16]. Moreover, there are important differences across disciplines in the way information is presented and represented and in their styles of reasoning, argumentation and inquiry that are best conveyed by teachers within the particular disciplines [14]. The problem that many readers share is not their inability to read the words on a page; instead, their most common problem is their inability to construct a coherent representation of the text meaning that is well integrated into their knowledge base. That is, they fail to deeply comprehend what they read [3].
These difficulties have an impact on science learning beyond the traditional reading activity. The landscape of science education is changing to incorporate disciplinary core ideas, cross-cutting concepts, and scientific practices as three different dimensions of learning [10]. The ability to read for understanding affects one’s ability to learn science ideas through the practices that rely heavily on comprehension of text, such as argumentation and modeling, as well as for theory development and dissemination, and even course communication. Thus, it is critical to address these issues for the future of science learning.
2.2 Computer Interventions in Reading Comprehension
To date, the What Works Clearinghouse has evaluated four computer-based literacy programs that had positive effects on comprehension at the middle- or high-school levels: AcceleratedReader [28], Fast ForWord [12], Read180 [20], ReadingPlus [8]. All four programs focus primarily on developing fluency and vocabulary in order to improve comprehension, and are designed to be embedded in language arts or special education classrooms. In contrast, eBRAVO targets deep levels of comprehension in students who are reasonably fluent readers in their science domain classroom. Decoding at a fifth- grade level or higher, these students lack the skills to deeply or even adequately comprehend informational texts.
Recent research supports the development of automated, computer-based interventions that teach students to utilize reading strategies to learn from challenging, complex texts [4, 17, 25, 26, 31]. A leading example developed by McNamara and colleagues called interactive Strategy Training for Active Reading and Thinking (iSTART) utilizes natural language processing algorithms to automatically evaluate and provide feedback on students’ written self-explanations of text (e.g., [25]). Two newer versions, iSTART extended practice and iSTART-ME, also allow teachers to upload texts [17, 18]. However, eBRAVO differs from other online and computer-based programs, such as iSTART, in that it offers a more comprehensive approach to comprehension instruction embedded in the progressive study of content course texts. eBRAVO addresses all levels of comprehension, targeting local and global coherence building, gistmaking and inferencing, as well as metacognitive and problem-solving strategies (e.g., self-explanation) that proficient readers employ to identify and remedy comprehension breakdowns. This comprehensive approach to instructing deep reading comprehension assists students in constructing a coherent textbase and a situation model with multiple connections to the personal knowledge base, thereby ensuring durable and accessible memory for the course content. Furthermore, The pedagogical components of eBRAVO are designed following the CI comprehension model [21, 22], a thoroughly researched theory that describes the cognitive processes that contribute to reading comprehension. eBRAVO focuses on instruction of literacy skills together with knowledge-building, using a gradated series of texts contextualized within a content area.
2.3 Intelligent Tutoring Systems and Video Engagement
Intelligent Tutoring Systems (ITS, [1, 15]) are a promising approach to supporting learning. These tools provide students with a cohesive experience in a digital learning environment, using their engagement with the problems or tasks in the system to develop an underlying learner model that represents their current understanding of the content as well as relevant situational factors such as engagement or affect [7, 11]. ITS then use this model of the learner’s mental state to determine the appropriate next action for developing deeper understanding. This approach has been shown to be as effective as one-on-one time with a tutor, providing students with individualized support that can outpace time on task in a larger classroom activity [30].
Videos are increasingly prevalent as a learning tool, but these are primarily studied at a postsecondary level for MOOC and Blended/Flipped Learning environments (see, e.g. [32]). That said, it is important in supporting reading comprehension as a skill in a computer & learner system; when you cannot rely on the learner’s deep comprehension of the written word, you have to rely on connections between written words, visual diagramming, and narration. This design consideration inspires the demonstration found in the example screen from the video lesson in Fig. 1.
However, when working with these ITS systems at the secondary level, many different paradigms of motivation are important. Many researches have cited issues with motivation in the use of ITS (e.g., [2]), and different approaches to improving engagement have been explored, including leveraging aspects of gaming directly in the interaction (e.g., [24]).
3 Research Context
Our eBRAVO project builds on BRAVO [5], a connected ecology curriculum with reading comprehension lessons originally designed for deployment in language arts classrooms. BRAVO operationalized the CI model of reading comprehension discussed in the background section above by incorporating five central lessons on the increasingly complex layers of the model, flowing from issues of local cohesion (“Linking Ideas” and “Getting the Gist”), global cohesion (“Summary” and “Graphic Organizers”) and situation modeling (“Inference”). These lessons were interspersed among the curriculum chapters, taught by the teacher to the whole class at once.
The eBRAVO curriculum takes the linear process of BRAVO and creates an adaptive reading experience on their web-enabled devices diagramed in Fig. 2 to work through an improved twelve chapter ecology curriculum. This is designed to allow students to encounter lessons as needed, rather than as a class. To accomodate this personalized learning experience in an intelligent tutoring approach, we adapted the five class-level lessons from BRAVO into videos and a practice problem personalized for each chapter, creating 119 lessons (chapter one does not have an inference lesson, since there is no assumed prior experience to draw upon).
As students work within eBRAVO in their browser, they first see a short chapter of 3 to 11 paragraphs of ecology reading content connected to what they have read so far in the curriculum. These readings incorporate in-text highlighting and clickable definitions for key vocabulary, along with embedded figures including images of important components or diagrams of certain systems. Once students have completed their readings, they proceed directly to an open-ended log asking them to record the big ideas found in the text. eBRAVO then probes their deep comprehension with multiple choice questions that target both issues brought up directly in the text as well as inferences and connections to greater ideas from previous texts. If the system determines a student should see a reading lesson based on their comprehension scores, it then gauges which type of reading support (linking ideas, gist, summary, graphic organizer, or inference) with which the student needs help and sends them to the appropriate lesson for that chapter and content, as seen in Fig. 1 above. These lessons incorporate two to five minute videos that outline the key comprehension strategy that is targeted and demonstrate how to apply this comprehension strategy to parts of the chapter that the student has just completed.
3.1 Data and Analysis
In this paper, we present results from 27 middle school students who provided family consent and used eBRAVO in a four-week summer intensive program. These students used eBRAVO in their 60-min class period four days a week over 12 class periods. We gave them the TORC paragraph reading activity [19] as a pre and post assessment activity in order to independently gauge reading comprehension ability. During this program, students were not assigned grades or other curriculum outcomes, so external motivation for performance was lower than usual for a typical classroom deployment.
We use a learning analytics [29] approach to understand student activity within eBRAVO. We focus our efforts on relating student engagement with the reading lessons as measured by their clickstream with the video content to the resulting outcomes on practice problems in the lesson as well as future questions in the curriculum. We use a feature driven sequential analysis approach [27], creating independent variables of video and lesson engagement. We explore the distribution of these engagement variables to understand different patterns of use.
4 Results
Overall, our student sample demonstrated a wide variety of lesson personalization within eBRAVO, seeing on average 5.44 (SD 3.59) lessons, with two students seeing 11 lessons (almost one lesson every chapter) and four students seeing no lessons.
Our primary division of students is based on patterns of engagement across videos, as seen in Fig. 3. Overall, we see very few students did not reach at least one lesson (No Lesson), a group of students maintained engagement with lessons (Engaged), and a small group of students simply did not engage (Unengaged). However, the largest group of students began the curriculum engaged, but did not maintain their video engagement by their final lesson (Dropoff). Though “Dropoff” students continued to struggle with the deep comprehension questions, they began to show attrition when addressing their issues with the lesson content.
These distinctions demonstrate interesting correlations with a variety of student outcomes. The first outcome of note is the student experience in terms of the average number of lessons encountered within each group, as seen in Fig. 4. The three groups that encountered lessons all demonstrated patterns that trend towards significance on a Kruskal Wallis H Test (p = 0.099), with students who demonstrate patterns of dropoff and unengagement trending towards increasing numbers of lessons seen.
These groupings also demonstrate interesting implications for demonstrated learning outcomes. In a pairwise t-test analysis of students’ comprehension scores within eBRAVO, a significant difference appeared when comparing “No Lesson” students to the “Dropoff” group (p = 0.0405), with the No Lesson students doing significantly better on in-system assessments. However, these results did not carry over to the TORC test at significance (p = 0.15).
5 Discussion
We see significant differences among each group in our breakdown.
5.1 No Lessons
For our “No Lessons” group, they clearly worked to ensure that they succeeded at the comprehension checks at the end of each chapter. This group demonstrated a mastery of the content within the system, but did not demonstrate a significant difference in the reading outcomes section, which may be due to causes discussed in the limitations section below.
5.2 Engaged
While our “Engaged” group demonstrated a willingness to keep with the program, and an ability to eventually master the abilities needed to complete the comprehension checks, they also did not demonstrate an improvement.
5.3 Dropoff
Our “Dropoff” group showed continued troubles with the comprehension checks within the curriculum - they are consistently diverted to lessons, which could be the source of their frustration and the reasoning behind their transition from engaged to unengaged.
5.4 Unengaged
The “Unengaged” group acted as you might expect - they never watched lesson videos, and they subsequently also never appeared to succeed on comprehension checks which meant they continued to be diverted to lessons. This did not connect to significant differences in in-system or pre/post test measures, which may be attributable to sample size as discussed in the limitations section below.
6 Conclusions
First and foremost, these results have implications for our design, suggesting the need to revisit our approach of repeated presentation of concepts. We found that students quickly found themselves breaking into different patterns of engagement that were easily distinguishable within the clickstreams and formed a distinct set of outcomes.
Moreover, these results should be addressed in other ITS tools that rely on a repeated sampling approach to concept understanding, demonstrating the need for engagement with lesson content to be motivated further. Students in this setting without external motivation from grades or other factors often do not continue to engage with the same content just because it is presented in a new context – they quickly grow tired of the approach.
6.1 Limitations
One important note to consider when discussing this work is the limited scope of this deployment. The first element of scope to consider is the use of 27 students under one teacher, which does not allow us to generalize much beyond the scope of that program. Many of our usage patters also resolved to very small numbers, which impacted our ability to look for significant differences, particularly in course outcome measures among groups such as the “No Lessons” (4 students) and “Unengaged” (3 students) groups. It will require a larger deployment to hone in on differences between these groups and gain further insight. Furthermore, it is important to remember that this was a deployment over a three-week summer program with no formal assessment policies, which did not allow for the teacher to implement any level of external motivation to participate with true fidelity. Many participants may have seen this as an opportunity to “slack off” when faced with learning from video lessons.
7 Future Work
These results and conclusions have motivated some significant changes in the workflow of the eBRAVO system. Some of our changes have already been implemented - we have focused on upgrading our teacher-facing dashboard to better provide a view for how the students are progressing through the materials within eBRAVO and what measures of success they are showing, in order for the teacher to provide just-in-time intervention to a student that may be struggling or showing signs of disengagement. These results have also prompted discussions of more significant redesign of the eBRAVO platform, incorporating intrinsic motivation patterns within the system, such as automated feedback and potential gamification [24] aspects.
References
Anderson, J.R., Boyle, C.F., Reiser, B.J.: Intelligent tutoring systems. Science 228(4698), 456–462 (1985)
Baker, R.S.: Modeling and understanding students’ off-task behavior in intelligent tutoring systems. In: Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 1059–1068 (2007)
Biancarosa, G., Snow, C.E.: Reading next: a vision for action and research in middle and high school literacy: a report from Carnegie Corporation of New York. Alliance for Excellent Education (2004)
Caccamise, D., Franzke, M., Eckhoff, A., Kintsch, E., Kintsch, W.: Guided practice in technology-based summary writing (2007)
Caccamise, D., Friend, A., Groneman, C., Littrell-Baez, M., Kintsch, E.: Teaching Struggling Middle School Readers to Comprehend Informational Text. International Society of the Learning Sciences, Boulder, CO (2014)
Chall, J.S.: Stages of reading development (1983)
Chaouachi, M., Frasson, C.: Mental workload, engagement and emotions: an exploratory study for intelligent tutoring systems. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 65–71. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30950-2_9
Clearinghouse, W.W.: Wwc intervention report: Reading plus® (2010). Accessed 10 April 2016
Council, N.R.: The nation’s report card (2017)
Council, N.R., et al.: A framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. National Academies Press, Washington (2012)
D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012)
Gillam, R.B., et al.: The efficacy of Fast ForWord language intervention in school-age children with language impairment: a randomized controlled trial. J. Speech Lang. Hear. Res. 51(1), 97–119 (2008). https://doi.org/10.1044/1092-4388(2008/007)
Goldman, S.R.: Discourse of learning and the learning of discourse. Discourse Processes 55(5–6), 434–453 (2018)
Goldman, S.R., et al.: Disciplinary literacies and learning to read for understanding: a conceptual framework for disciplinary literacy. Educ. Psychol. 51(2), 219–246 (2016)
Graesser, A.C., Conley, M.W., Olney, A.: Intelligent tutoring systems (2012)
Heller, R., Greenleaf, C.L.: Literacy instruction in the content areas: getting to the core of middle and high school improvement. Alliance for Excellent Education (2007)
Jackson, G.T., Boonthum, C., McNAMARA, D.S.: iSTART-ME: situating extended learning within a game-based environment. In: Proceedings of the workshop on intelligent educational games at the 14th annual conference on artificial intelligence in education, pp. 59–68. AIED Brighton (2009)
Jackson, G.T., Guess, R.H., McNamara, D.S.: Assessing cognitively complex strategy use in an untrained domain. Top Cogn. Sci. 2(1), 127–137 (2010)
Jongsma, E.A.: Test review: test of reading comprehension (TORC). Read. Teach. 33(6), 703–708 (1980)
Kim, J.S., Capotosto, L., Hartry, A., Fitzgerald, R.: Can a mixed-method literacy intervention improve the reading achievement of low-performing elementary school students in an after-school program? results from a randomized controlled trial of read 180 enterprise. Educ. Eval. Policy Anal. 33(2), 183–201 (2011)
Kintsch, W., Kintsch, E.: Comprehension. In: Children’s reading comprehension and assessment, pp. 89–110. Routledge (2005)
Kintsch, W., Walter Kintsch, C.: Comprehension: A Paradigm for Cognition. Cambridge University Press, Cambridge (1998)
Lee, O., Quinn, H., Valdés, G.: Science and language for english language learners in relation to next generation science standards and with implications for common core state standards for english language arts and mathematics. Educ. Res. 42(4), 223–233 (2013)
McNamara, D.S., Jackson, G.T., Graesser, A.: Intelligent tutoring and games (ITAG). In: Gaming for classroom-based learning: Digital role playing as a motivator of study, pp. 44–65. IGI Global (2010)
McNamara, D.S., Levinstein, I.B., Boonthum, C.: istart: Interactive strategy training for active reading and thinking. Behav. Res. Methods Instrum. Comput. 36(2), 222–233 (2004)
Meyer, B.J., Wijekumar, K.: A web-based tutoring system for the structure strategy: theoretical background, design, and findings. In: Reading Comprehension Strategies: Theories, Interventions, and Technologies, pp. 347–375 (2007)
Quigley, D., Ostwald, J., Sumner, T.: Scientific modeling: using learning analytics to examine student practices and classroom variation. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, pp. 329–338 (2017)
What Works Clearinghouse: Accelerated Reader\(^\text{TM}\). What Works Clearinghouse Intervention Report. Updated. Institute of Educational Sciences (2016)
Siemens, G., Gašević, D.: Special issue on learning and knowledge analytics. Educ. Technol. Soc. 15(3), 1–163 (2012)
VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)
Wijekumar, K.K., Meyer, B.J., Lei, P.: Large-scale randomized controlled trial with 4th graders using intelligent tutoring of the structure strategy to improve nonfiction reading comprehension. Educ. Technol. Res. Dev. 60(6), 987–1013 (2012)
Yousef, A.M.F., Chatti, M.A., Schroeder, U.: The state of video-based learning: a review and future perspectives. Int. J. Adv. Life Sci. 6(3/4), 122–135 (2014)
Acknowledgements
The research reported here was supported, in whole or in part, by the Institute of Education Sciences, U.S. Department of Education, through grant R305A170142 to the University of Colorado Boulder. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2020 The Author(s)
About this paper
Cite this paper
Quigley, D., Caccamise, D., Weatherley, J., Foltz, P. (2020). Exploring Video Engagement in an Intelligent Tutoring System. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2020. Lecture Notes in Computer Science(), vol 12214. Springer, Cham. https://doi.org/10.1007/978-3-030-50788-6_38
Download citation
DOI: https://doi.org/10.1007/978-3-030-50788-6_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50787-9
Online ISBN: 978-3-030-50788-6
eBook Packages: Computer ScienceComputer Science (R0)