1 Introduction

According to the 2013 National Longitudinal Transition Study-2 (NLTS-2) report by the US Department of Education [1], students without disabilities are approximately three times as likely as students with disabilities to earn high GPAs; 20 percent of students in the general population earned GPAs of 3.35 or above, compared with 6 percent of students with disabilities [1]. The poor performance in their academic courses is a major factor in failing to graduate and acquiring the skills that are desired by employers. The employment data collected by the American Foundation for the Blind (AFB) between May 2016 and April 2017 show a 70% unemployment rate for people who are blind or have vision impairment [2].

Mathematics is a foundation for students to learn and persevere in social and professional worlds. Recent advancements in Information and Communication Technologies (ICT) have provided students with many virtual tools to improve their knowledge of mathematics. Auditory assistive technologies have enabled digital access to math formulae on the web [7, 13, 24] and incorporated with multimodal applications; auditory methods have helped many visually impaired students learn math concepts [5, 20]. However, learning through listening, while continuing to be recognized as part of the expanded core curriculum for educating blind and visually impaired students, receives less attention [13]. There is a need to combine these techniques, such as learning to interact with the system through voice, learning by listening, and self-directed learning to create a barrier-free and accessible system.

This research focuses on developing MyAccessible Math, a multimodal platform for Students with Vision Impairments (SVIs) to practice math concepts [17]. We developed the prototype with the following features: first, provide an easy-to-learn interaction mechanism to the students for clear communication with the application; second, implement an algorithm that solves the math problems and provides step-by-step solutions to students; finally, integrate a highly interactive text-to-speech library that gives speech control to the student.

The goal of the project is to promote self-learning for SVIs with a specific focus on algebra. Within the algebra curriculum, solving linear equations is one of the foundational topics in which students make the transition from reasoning with numbers to reasoning with unknowns [10, 23]. We designed MyAccessible Math using an iterative development process that involved feedback from students and their educators from the Alabama Institute for the Deaf and Blind (AIDB) and the Alabama School for the Blind (ASB) to ensure usability.

We investigate the application’s capability to support self-learning for SVIs using experimental research designs. Ten students from AIDB and ASB participated in assessing how comfortable and adequate is the interaction with the application. The research questions investigated by this study are:

  • How well does MyAccessible Math help students practice math questions?

  • What are the potential benefits and limitations in learning and practicing approaches used in MyAccessible Math?

  • How organized and easy was it to follow the content on the MyAccessible Math web portal for SVIs?

This study is one step in the sequence of future evaluations to improve the mathematics curriculum and self-learning for SVIs.

2 Assistive tools for mathematics for SVIs

With the increased interest in digital learning over the past decade, several authors have conducted systematic literature reviews documenting assistive technologies implemented for visually impaired people. These existing systematic reviews cover the effects of computer technology on school students in mathematics learning [19], ICT applications for visually impaired people [6], or digital learning technology for visually impaired students [3, 18].

Dumkasem [8] introduced a cloud-based mobile application, EyeMath. EyeMath allows users to upload images of a page snippet containing math expressions and converts them into sentences for the device’s Screen Reader (SR) program to read aloud. First, the application separates math expressions from plain text. Math expressions are then processed into an Abstract Syntax Tree (AST) and parsed into Thai sentences [8]. EyeMath uses Tesseract Optical Character Recognition (OCR) to extract text from images. The usability testing with five visually impaired people confirms the application’s potential benefits of reading math expressions aloud from images.

While tactile methods fail to accommodate students with graphs and visual cues, sonification is an effective way of conveying them [22]. AudioFunctions.web [4] is a web application that uses sonification, earcons, and speech synthesis to enable blind people to explore mathematical function graphs [4]. Based on user needs and preferences, AudioFunctions.web allows a global overview and analytical exploration of a given function graph on mobile and traditional devices through different interfaces such as touchpad, mouse, or keyboard. An experimental evaluation with 13 visually impaired participants shows that the proposed interaction methods are highly usable.

More recent technologies have attempted to develop multimodal systems for students to learn and practice mathematics. Elkabani [9] designed a framework that enables visually impaired students to learn and practice algebra like sighted students. The process is divided into four steps. The teacher provides algebraic expressions using a math editor. The system then converts math expressions to MathML objects. The interactive workspace in the system allows users to interact and navigate within questions using keys. The system provides auditory feedback in English and Arabic using the Text-to-Speech (TTS) engine. The workspace generates MathML objects from solutions submitted by students. Finally, teachers can access these solutions in MathML using a MathML editor. The systematic evaluation from a group of visually impaired individuals showed that the system increased their performance. It enables students to solve linear algebra exercises faster than Braille writers and allows them to study linear algebra without needing a sighted tutor.

Grossman [11] developed an automated text-based tutor to promote online math education. The system is built as a chatbot, presenting the math materials through a simple text-based interface. The chatbot uses informal languages, including emojis, to give a human touch to the application. The prompts include personalized feedback and answers to open-ended questions. In the first study, the authors examined the user preferences of 116 users, comparing MathBot with videos and written tutorials from Khan Academy. 42% of users preferred MathBot over videos from Khan Academy. The study concludes that conversational agents are promising tools in online education. However, the application is not designed considering students with disabilities. It requires users to use the on-screen keyboard to communicate with the chatbot.

These existing technologies facilitate SVIs and provide digital access to math formulae on the web and in Portable Document Format (PDF), use sonification and verbal messages to convey the overall structure of the graph with additional information, and incorporate auditory methods to provide hints when necessary. On the other hand, MyAccessible Math is a web application with the following features that differentiate it from prior work:

  • It is designed as a platform-independent and free-to-use web application that anyone can access from a personal computer, a tablet, or a mobile device;

  • It enables voice and keyboard navigation on the web and supports human-like interaction with the system using a voice-based assistant;

  • When asked, it provides step-by-step instructions and helps students solve and simplify math equations. It also enables arrow keys navigation within math questions.

This paper is organized as follows. Section 3 describes the design principles and technical implementations behind MyAccessible Math. Section 4 contains the experimental research study with ten visually impaired school students. Section 5 discusses results and suggestions from students and educators. The paper is concluded in Sect. 6 proposing future improvements.

3 Initial system design: MyAccessible Math v1

Fig. 1
figure 1

MyAccessible Math system architecture

Full size image

User-Centered Design (UCD) is an iterative design framework for creating easy-to-use and interactive self-learning platforms, especially for SVIs. As the name suggests, a UCD is built on the idea that the needs and preferences of end-users should drive the design process for new technologies [21]. The development of MyAccessible Math employed the UCD approach, which centered around the following key principles [12]:

  • User focus and active user involvement: The authors gained a comprehensive understanding of the target users through various research methods, such as interviews, surveys, and observations;

  • Iterative system design: Throughout the development process, accessibility features in the prototype were continuously refined and improved based on feedback received from users;

  • User testing: Usability testing was conducted to gather feedback from users, which helped identify potential usability issues and areas for improvement;

  • Contextual design: The design process took into account the context in which users would interact with the application. The application was adapted to fit their environment and workflow, ensuring a more personalized user experience.

MyAccessible Math is implemented using Java Servlets and Model-View-Controller (MVC) that eliminate the need for SRs by integrating text-to-speech and speech recognition JavaScript libraries. Figure 1 shows the system architecture of MyAccessible Math. The components as shown in Fig. 1 are explained below:

  • Model: The Model represents the application’s data and business logic. It is responsible for managing data storage, retrieval, and manipulation. In the context of MyAccessible Math, Questions and Topics are two of the Model classes that interact with the database;

  • View: The View is responsible for the presentation layer of the application, which is the user interface. Unlike traditional websites, MyAccessible Math uses text-to-speech, voice, and keyboard JavaScript libraries for user interaction;

  • Controller: The Controller acts as an intermediary between the Model and the View. The MyAccessible Math Controller receives voice or keyboard interaction requests from the View and performs necessary data manipulation using Math Solver algorithms. After processing the request, the Controller sends the data to the appropriate View and waits for further user interaction. Figure 2 shows the example user interaction for MyAccessible Math.

Fig. 2
figure 2

MyAccessible Math example user interaction

Full size image

3.1 Text-to-speech

MyAccessible Math uses ResponsiveVoice.jsFootnote 1 for two features. First, it plays a welcome message for users when accessing the page for the first time. Second, it supports human-like communication when incorporated with the speech recognition library to respond to students’ questions.

3.2 Speech recognition

Annyang.jsFootnote 2 is a free-to-use and lightweight speech recognition library that supports adding speech commands to the web application. MyAccessible Math uses Annyang for voice-enabled navigation and gives speech control to the user. These pre-defined voice commands are programmed with the application, and more commands can be added to extend the scope of the application.

3.3 Keyboard interaction

After the initial meetings with the educators at the AIDB, keyboard shortcuts were programmed in the application to support keyboard navigation when speech recognition fails. Jwerty.jsFootnote 3 is an open-source library that supports handling keyboard events on the web. Table 1 shows a list of a few voice and keyboard commands supported on the application.

3.4 Step-by-step solution

The practice module is responsible for helping students practice math concepts. This process is divided into four steps: The first step is to retrieve a brief description of the topic from the database and provide it to the student. The questions are stored in a secure MySQL database. The second step is to select questions one at a time for the chosen math topic from the database. Then, the math parser will take the question as an input and determines the steps to find the solution. Once the student is ready to practice questions, the student will be redirected to the practice questions page. If the student encounters any issues, the student can say terms such as ”next hint” or ”repeat question.”

$$\begin{aligned} \frac{5}{12} + \frac{1}{8} \end{aligned}$$
(1)

The current version of the math parser supports the following math concepts: place values, basic arithmetic, simplifying fractions, and solving linear equations.

One example of simplifying fractions with the step-by-step solution is shown below in Eq. (1) and Table 2.

Table 1 List of voice commands and keyboard keys
Full size table
Table 2 Steps to simplify fractions
Full size table

4 Research study

4.1 Initial meetings

The pre-evaluation meetings were focused on identifying the technological needs of SVIs and their educators. We presented the prototype of the MyAccessible Math to educators at the AIDB. We consulted six individuals, including assistive technology experts and instructors from the AIDB. These sessions were conducted at Auburn University and the E. H Gentry campus at the AIDB. The researcher gave demonstrations of the technologies used in MyAccessible Math. The meetings included questions related to recent trends in assistive technologies for math education for SVIs. Common concerns include: (1) the cost of high-tech devices, and (2) cross-compatibility problems caused by the many types of assistive devices and their various operating systems.

The meetings concluded with identifying three further improvements with the application for SVIs. First, provide a way for users to navigate inside a math question. Second, add a live speech-to-text conversion module for researchers to investigate issues related to speech recognition. Third, integrate keyboard navigation using shortcut keys if speech recognition fails. The educators recommended using basic math questions and asked to add support for elementary, middle, and high school SVIs. We chose to remodel the system based on information gathered from the interviews and a systematic literature review of assistive technology platforms.

4.2 Participants

Ten SVIs participated in this study. Students were selected based on: (1) a disability of visual impairment, including low vision or blindness; (2) enrollment in elementary, middle, or high school at the AIDB or ASB, where the study was conducted; and (3) willingness to participate (informed consent). Out of ten students, three were blind, and seven had moderate to severe vision impairment.

4.3 Setting and materials

This study was carried out at the AIDB on the E. H. gentry campus in the Assistive Technology laboratories. The Assistive Technology laboratory consisted of desks formed into a U-shaped sitting arrangement where the researcher had easy access to each student. Students worked individually with a member of the research team.

Materials included a computer and a headphone. The AIDB provided laptops with a built-in microphone and Google Chrome web browser for students to practice questions. At the end of the evaluation, students anonymously provided post-survey feedback. The post-survey questions are listed in Table 3.

Table 3 Post-survey questions
Full size table

4.4 Evaluation

We presented students with five math questions based on the difficulty levels for the evaluation. First, we described MyAccessible Math. Second, we demonstrated the accessibility features of the application. Students begin the assessment with the application by completing a short tutorial to familiarize themselves with the web application interaction. They learn to press “1” to learn shortcut keys, “4” to practice questions, “0” to skip the question, “7” to hear the previous hint, “8” to hear the next hint, and so forth. The application does not allow revisiting completed or skipped problems and does not keep track of students’ activity. We asked students to provide anonymous feedback on the application’s accessibility and interaction features. We asked students questions about their liked, disliked features, and future improvements.

5 Results and discussion

This paper shows the development and evaluation of MyAccessible Math conducted with 10 SVIs in Alabama. The participants used keyboard and voice navigation to practice math questions accurately. Educators at the AIDB provided math questions for the study. These questions ranged from simple arithmetic operations to solving linear equations. Participants voluntarily chose options to evaluate the elementary, middle, or high school modules. After the evaluations, we interviewed participants to assess web accessibility. Moreover, the students provided information about their experience using the application and how to improve the accessibility features for users who have low vision or are blind. Participants’ information was kept confidential.

Fig. 3
figure 3

Students’ responses for post-survey questions

Full size image

To evaluate the accessibility features on MyAccessible Math, we used a 3-point Likert scale. Figure 3 shows the post-survey responses from students.

5.1 Voice and keyboard interaction

Of the 10 participants, two participants stated that they had trouble understanding the UK English. Three participants mentioned that the speech rate was accurate and easy to understand. According to one student,

“I liked how the application spoke clearly and was not a robot voice I could not understand.”

Seven students liked keyboard interaction, which helped them a lot while navigating the web application. A student commented,

“The keyboard interaction was not frustrating; just press a key and move on.”

Four students mentioned how easy it was to use the application and liked the feature that the application would talk back and forth. One low vision student preferred the magnification on the mouse hover feature. Three students appreciated the part about using arrow keys to navigate inside a math question and how it would say each character out loud.

5.2 Educators’ feedback

Based on observations and experience working with SVIs over the users, educators provided valuable feedback about the application’s accessibility features. For the evaluation, educators provided the list of questions but requested to have the ability to add questions for future evaluations. The current version of the application is not accessible for visually impaired instructors. According to a visually impaired instructor at the AIDB, having a Braille file upload module to add questions would be an excellent application feature.

5.3 Limitations

The students suggested various features they liked, disliked, and suggested further improvements in the application. Two students had trouble using the microphone and used keyboard interaction for the study duration. Since ten participants were situated in an enclosed room during the evaluation, some participants had difficulty using voice interaction due to voice mix-ups. Four students liked the application and suggested adding more math concepts. Two participants did not like the inability to add answers. Out of 10 participants, five participants did not have a preference or liked the black-on-white color theme of the application. Two low-vision participants did not like the color theme and preferred to have a darker theme. Five participants suggested adding an option to have a drop-down menu to choose the preferred color theme. Since participants have experience using NonVisual Desktop Access (NVDA), they suggested having shortcut keys consistent with the NVDA screen reader. For example, “CTRL” should be pressed to pause the speech instead of “5.” Two elementary school students faced trouble locating shortcut keys on the keyboard.

6 Conclusion and future work

The prototype presented in this paper was developed with two goals. First, to provide an easy-to-learn interaction mechanism to the students for clear communication with the application and integrate a highly interactive text-to-speech library that gives speech control to the student. We evaluated the interactive features of the web application using an experimental research study. Results show that students favored voice-based interaction over keyboard interaction. Second, the application uses a text-to-speech and a speech recognition JavaScript library that gives speech control to the user. This approach enables communication with the system but limits the responses to closed-ended questions. The researcher must manually provide the questions and answers for the system to work as intended. The system is designed to work for visually impaired students where clear communication is a key; this was a severe downside. To improve human–computer interaction, we plan to integrate an intelligent conversational bot.

The prototype uses an algorithm to parse and evaluate basic arithmetic questions and linear equations. We propose introducing an expression parser and evaluator that provides step-by-step solutions to a wide range of complex mathematical concepts. With further development and evaluations from SVIs, we hope to extend MyAccessible Math and enable support for parent–teacher and student–student collaboration.