Keywords

1 Introduction

ITS continue to be widely used in educational settings and have been shown to lead to substantial learning gains, including in mathematics [1]. However, the use of this technology may have differential effects. In classrooms across the United States, students with disabilities are co-located with their non-disabled peers. When teachers use educational technology that is not accessible to students with disabilities, it can leave these students confused, isolated, and often trailing in progress compared to their classmates. This can be very detrimental to the learning environment. Accessible alternatives to most ITS are nonexistent resulting in scant research on accessible ITS.

While such work is long overdue, we recognize the myriad of challenges. These include that researchers may not be aware of detailed student characteristics, sample sizes may be too small to draw definitive conclusions, different adaptations to ITS may be needed to support students with different kinds of disabilities, etc. However, we believe that many of the required pieces to build accessible ITS already exist. Our work is a modest first step in putting them together to create assessable mathematics ITS for students who are blind or have motor-function disabilities.

2 Prior Work

There are many ways to author ITS. The one relevant to our work is Cognitive Tutor Authoring Tools (CTAT) integrated through TutorShop to create an example-tracing tutor. Development costs of example-tracing tutors using CTAT are significantly lowered compared to “historical” estimates for ITS while still allowing for sophisticated tutoring behaviors [2]. These include providing step-by-step guidance on complex problems while recognizing multiple student strategies while recording action-level data. CTAT has been shown to be quite general, with tutors built for many domains covering a range of pedagogical approaches, including collaborative learning [3]. Tutors have also seen classroom use, including the Mathtutor [4].

Another tool relevant to creating our accessible ITS is Bazaar. Bazaar is a publicly available architecture for orchestrating conversational agent-based support and is intended to facilitate research on collaborative learning. It hosts a library of reusable behavioral components that each trigger a simple form of support. More complex supportive interventions are constructed by orchestrating multiple simple behaviors. Its flexibility means it can be used to develop platforms very rapidly for investigating a wide range of dynamic support research questions [5]. We had to adapt this tool to our objectives, i.e., only one user instead of multiple users.

There is scant research on developing ITS for blind students or those with motor-function impairments. Regarding mathematics education for visually impaired students, a primary obstacle is an inherent difficulty in managing structural information included in math formulae [6], and audio-based interactive computer interfaces have been shown to enhance learning and cognition [7]. We build upon this prior work to create and implement a framework for authoring and researching accessible ITS.

3 Framework

Our framework is intended to structure the development of accessible ITS which can be used by students with motor-function disabilities and blind students. Since these groups require different kinds of support, our tutor has two layers of adaptivity. Specifically, one ITS interface is speech-enabled, while the other embeds this interface into a platform containing a conversational agent which is text and speech generating.

The speech-enabled ITS (Fig. 1, left) is intended to support students with motor-function impairments. It allows users to use speech to navigate the tutor interface by saying which input field should be brought into focus by stating that field’s label. It also allows common ITS functions such as using speech to enter values, ask for hints, finish a problem, etc. This ITS has been pilot tested but is not yet classroom ready.

The speech-enabled ITS embedded in a conversational agent (Fig. 1, right) is intended to support students who are blind. The primary impetus behind the text generation facet of the conversational agent is that most CTAT components are not navigable by assistive technologies. Our platform allows screen-readers to read the conversational agent’s text output which can also be converted to speech. As described below, this ITS with a conversational agent has not yet been pilot tested.

Fig. 1.
figure 1

Left pane: Speech-enabled example-tracing tutor interface created using CTAT and Web Speech API; (1) Speech-enabled button to begin and end tutor interaction, (2) each input field has distinctive labels for speech navigation, and (3) in-focus input field is surrounded by hue. Right pane: Speech-enabled ITS embedded in conversational agent created using Bazaar.

Full size image

4 Development Progress and Insights

530 students enrolled in 6 courses taught by one instructor at a large public university participated in this study during the 2021 calendar year. First, 115 students in two introductory college statistics courses during the Spring 2021 semester completed a set of 25 questions on normal random variables (NRV) assigned as homework immediately after discussing that topic in class. The homework used an ITS created using CTAT integrated into the course through TutorShop [2]. We used the ‘Stat 1222–004 S21 Chap 5 HW’ and ‘Stat 1220–010 S21 Chap 5 HW’ datasets accessed via DataShop [8] to assess the efficacy of this tutor and observe student behaviors with standard features available in CTAT. Subsequently, the speech-enabled components were added to the tutor interface. Specifically, we added the Web Speech API and tested the ITS with two volunteers, both of whom provided very positive feedback.

Additionally, 118 students in a different introductory college statistics course at the same institution completed 70 questions outside of an ITS over the course of the semester. Each question consisted of two parts. The first part required students to show their work while solving a question relevant to the material recently covered in class. The second part required students to describe the steps they had taken in the first part. This resulted in over 20,000 explanations of individual solution steps. We are using these explanations to assess the verbally described actions students may take in the ITS embedded with the conversational agent, and author supports that are useful to students.

A significant insight we have gained in this work is the myriad of ways students describe their solution steps. These have ranged from elaborate sentences to a single word, oftentimes omitting or misusing mathematics terminology which significantly affected the accuracy of our conversational agent. To mitigate this, during the Fall 2021 academic year, we collected an additional 55,000 solutions and explanations on 76 problems generated by 297 students. We are incorporating these into our model.

5 Conclusion, Limitations, and Future Research

Our work is a modest first step in addressing the need for accessible ITS and filling the void of research in this area. Nevertheless, we make several noteworthy contributions. First, we provide a framework on how to author accessible ITSs using existing tools, namely CTAT and Bazaar. While modifications had to be made to both, the modifications were modest and pilot testing showed promising results. Furthermore, we piloted an ITS with 115 students in two introductory college statistics courses to establish a reference for actions within that ITS and collected over 75,000 solutions steps and explanations by 415 students on 146 questions outside of an ITS to create support within an accessible ITS. Nevertheless, our work as of yet has its limitations. Chief among these is that the system has not yet been used with the target population.

While we already have rich data, once our ITS is utilized in vivo we will have to contend with such issues as analyzing concurrent but different data streams. Furthermore, the use of verbal protocol can give us insights that written responses potentially lack. Additionally, we will be able to compare the impact on mathematics learning of our platform to that of other platforms, while accounting for the fact that some students may have a disability. We will have to grapple with design issues, such as how best to incorporate multiple input streams (speech, braille, and QWERTY keyboards) into cogent responses from the combined ITS/conversational agent platform to the user. The platform’s responses also could take the form of speech and text generation, refreshable braille display output, or visual output on a computer screen.

Lastly, we are excited to take these initial steps in making accessible ITS and break ground on research on this topic.