Keywords

1 Introduction

In recent years, information and communication technology infrastructure has improved in schools and e-learning has become increasingly popular. One of the most important functions of e-learning is automatic assessment to evaluate a student’s understanding of course content. Computer-aided assessment (CAA) is an old technique and was applied to many subjects, even before learning management systems (LMS) became popular. One of the most common question types in CAA systems is multiple-choice questions (MCQ), wherein the potential answers are provided by a teacher and students select a single response as their answer. A well-constructed MCQ provides a correct answer with plausible distracters, which are usually decided by knowledge of common student errors.

CAA has been successfully used in language education. For example, CAA has been applied to placement tests based on Item Response Theory and computer adaptive testing has been compared with pencil and paper testing in terms of validity and efficiency (Koyama & Akiyama, 2009).

CAA has also been carried out using MCQs in scientific subjects, but the MCQ format is not sufficient to evaluate a student’s comprehension level. For example, when students answer MCQ type questions, students can simply choose an answer from a list even if they do not know the correct answer, and there is a possibility of answering correctly by guessing. In order to avoid these problems with MCQs, it is preferable to adopt a question format wherein students provide answers containing mathematical expressions, which are subsequently evaluated. There are some systems that evaluate student-provided answers. In one of the major systems, CIST-Solomon (CIST, 2016; Komatsugawa, 2004), which has more than 30,000 subject areas including mathematics, students construct mathematical expressions using software such as Flash.

In this study, we focus on a CAA system where students provide mathematical expressions through keyboard input and the answer is evaluated using a computer algebra system (CAS). For example, for the differentiation question \(\frac{d}{dx}\left( {\frac{1}{4}x^{2} + \frac{1}{2}x + 1} \right)\), the correct answer is \(\frac{1}{2}\left( {x + 1} \right),\) but some students provide \(\frac{1}{2}x + \frac{1}{2}\), others provide \(\frac{x + 1}{2}\), etc. These answers are all mathematically equivalent and correct, and the evaluation of equivalence is underpinned by CAS. Recently, this kind of math e-learning system has become popular and, to the best of our knowledge, there are three main systems being used in Japan: MATH ON WEB, STACK, and Maple T.A. Although the importance of questions with student-provided answers is understood in e-learning of mathematics, many teachers are now seeking effective ways to carry out mathematics e-learning using CAA. In this paper, we summarize the utilization of math e-learning systems in Japan, present some associated problems, and propose solutions to a variety of issues with implementing these technologies.

2 Math E-Learning Systems in Japan

Hereafter, we focus on online assessment systems for mathematics using CAS as a mathematics e-learning system. In contrast to multiple-choice or true-or-false choice questions, which are referred to as teacher-provided-answer questions, online assessment for mathematics provides student-provided-answer questions, wherein students submit numerical values or mathematical expressions as answers to the questions. In order to assess answers to student-provided-answer questions, math e-learning systems evaluate these answers using CAS. In this section, three math e-learning systems are briefly discussed: MATH ON WEB, STACK, and Maple T.A. These math e-learning systems, which are representative of the systems in use in Japan, use Mathematica, Maxima, and Maple, respectively, as their CAS to evaluate student answers. The problems associated with the continued use of these systems are outlined at the end of this section.

2.1 MATH ON WEB

E-learning/e-assessment systems based on webMathematica have been developed and are currently being used in mathematics education for first-year students at Osaka Prefecture University (2016; Kawazoe, Takahashi, & Yoshitomi, 2013; Kawazoe & Yoshitomi, 2016a, b). The systems are available on the ‘MATH ON WEB’ website. The website has two systems: web-based mathematics learning system (WMLS) and web-based assessment system of mathematics (WASM).

WMLS is a self-learning system aimed at promoting students’ after-class learning. It has two sections: drill section and simulation section. The drill section offers an online mathematics exercise environment with more than 1000 mathematics problems in calculus and linear algebra courses for first-year university students. When a student submits his/her answer to a presented problem, the system analyses the answer using Mathematica and provides a feedback message. When the answer is incorrect, the system provides a different feedback message according to the error type identified. The simulation section offers simulation type content that assists students in learning mathematical concepts.

WASM is an online mathematics assessment system with two different modes: assessment mode and exercise mode. The basic mechanism of WASM is the same as the drill mode of WMLS, but in WASM, problems are presented in random order or are randomly generated by Mathematica. In the assessment mode, online assessment tests are classified with respect to learning units and achievement levels, and students can assess their achievement in each learning unit using the online test associated with the learning unit. In the exercise mode, students can carry out problem-solving exercises in each assessment test. WASM has various improvements over WMLS. One such improvement is the implementation of popup keyboards based on jQuery Keyboard (Wood, 2014), which enables students to input their answers (Kawazoe & Yoshitomi, 2016b).

At Osaka Prefecture University, more than 600 students use the systems annually and both systems are used to promote students’ after-class learning and the implementation of blended learning environments (Kawazoe & Yoshitomi, 2016a). Quantitative analysis of the log data in WMLS (Kawazoe et al., 2013) showed that students use the system mainly in the late afternoons and at nights. Hence, it can be concluded that WMLS promotes students’ after-class engagement. Statistical analysis of engineering students (ibid.) shows that there is a positive correlation between the frequency of use of the system and achievement in mathematics. Kawazoe and Yoshitomi (2016b) reported on a blended learning linear algebra class with WMLS and WASM for first-year engineering students and noted that many students stated that the blended learning approach is useful and preferable.

The next objective of the MATH ON WEB project is to develop a Moodle plugin for WMLS and WASM. The development of the Moodle plugin is still underway, but a prototype has already been developed (Nakahara, Yoshitomi, & Kawazoe, 2016).

2.2 STACK

STACK, developed by Sangwin (2013), uses Maxima as its CAS to evaluate students’ answers. STACK not only assesses the mathematical equivalence of students’ answers but also generates outcomes, such as providing feedback according to the mathematical properties of students’ answers. The feedback function is implemented using the potential response tree (PRT) mechanism. PRT is an algorithm that establishes the mathematical properties of students’ answers and provides feedback specifically to each student.

STACK is being utilised for several subjects in many institutions in Japan. Taniguchi, Udagawa, Nakamura, and Nakahara (2015), who used STACK in a math class at Nihon University, used logistic regression analyses to document that STACK is effective. STACK is also being used in physics classes at Toyama University and Nagoya University. Basic support for scientific units has been added to the latest version of STACK; this function is expected to enhance the use of STACK in physics class.

STACK is an open source system and users can develop required functions. For example, the plot function in STACK is poor. Specifically, only the drawing of single variable functions is supported. The plot function has been enhanced using Maple (Nakamura, Amano, & Nakahara, 2011) and gnuplot (Fukazawa & Nakamura, 2016).

2.3 Maple T.A.

Maple T.A. is a web-based online testing and assessment system developed by Maplesoft, a Canadian software company. It was designed especially for science, technology, engineering, and mathematics (STEM) courses. Further, it offers various question types, flexible assignment properties, full-featured gradebook with reporting and analytical tools, seamless connectivity to any LMS, and support for multiple languages. In recent years, Maple T.A. has been gradually and steadily adopted for STEM education in academic institutions such as high schools, colleges, and universities in Canada, U.S.A., many European countries, China, and Taiwan (Maplesoft, 2016). After adopting and utilizing Maple T.A., they not only successfully reduced their grading burden but also improved the STEM education learning environment, resulting in students being strongly engaged in the courses. Consequently, they also view Maple T.A. as an important teaching tool.

By contrast, in Japan, Maple T.A. has been promoted by its distributor Cybernet Systems, a Japanese software company, for several years and is gradually beginning to be recognized as an online testing and assessment system for STEM education. Cybernet Systems already has a user—the Faculty of Science and Technology at Ryukoku University. The faculty utilizes Maple T.A. for fundamental mathematics education such as pre-entrance education and remedial education, bridging the gap between high schools and university. They state that Maple T.A. is capable of improving and enhancing the basic math ability of students and thus, they have plans to expand the use of Maple T.A. to a wide variety of STEM subjects in the future (Higuchi, 2015).

Cybernet Systems experimentally introduced Maple T.A. in a STEM programming course at Gakushuin University, and also assessed the functionality and performance of Maple T.A. (Cybernet Systems, 2012). The course provided students with a small test designed specifically to measure the programming skills of Maple. Eventually, all the results associated with this course are managed within Maple T.A. along with all the results of external items such as offline assignments graded using a Maple T.A. rubric.

In 2016, Cybernet Systems launched and conducted a project aimed at evaluating the capabilities and performance of Maple T.A. in seminars and classes with students, and obtained the cooperation of math instructors at universities. Six math instructors from different universities joined the project and evaluated Maple T.A. in their own math seminars and classes, with capacity ranging from several students to more than 200 students. Their main objective was to ascertain whether Maple T.A. enabled them to efficiently streamline the existing grading workflow and reduce the workload associated with creating questions, marking tests, and managing gradebooks. The instructors were satisfied with Maple T.A to some extent. Furthermore, Cybernet Systems obtained diverse feedback from the instructors and students, which centered around two major issues. One was a need to expand the math content from the viewpoint of the instructors; the other was a need to improve the math input interface from the viewpoint of the students.

2.4 Common Challenge

E-learning systems, especially CAS-based mathematics e-learning systems, are undoubtedly powerful tools for developing effective mathematics learning environments. However, CAS-based e-learning systems, including the aforementioned three systems, have two issues in common: inconvenience in inputting answers and heavy workload in developing content.

In CAS-based mathematics e-learning systems, students have to input their answers using the input form provided by CAS, which students consider inconvenient. Such inconvenience should be resolved; hence, developing effective math input interfaces for these systems is important. We discuss this issue in detail in Sect. 8.3.

The issue of heavy content-development workload originates from the fact that, in many Japanese universities, e-learning content is usually developed by teachers—specifically, a very limited number of teachers. If the teachers could share the content or resources beyond system differences, their burdens would be reduced. We investigate this problem in Sect. 8.4 and we will determine the importance of building a common base for creating questions in Sect. 8.5.

If the aforementioned two issues can be overcome, the use of e-learning systems in university mathematics education would become more pervasive.

3 Math Input Interfaces

As described at the end of the previous section, one of the problems associated with math e-learning systems is the math input complexity for questions requiring the input of mathematical expressions as answers rather than multiple-selection or number input types of answers. For example, in order to input \(3x^{2} - \frac{2x}{{\left( {x^{2} + 1} \right)^{2} }}\), which is an answer for the differentiation \(\frac{d}{dx}\left( {x^{3} + \frac{1}{{x^{2} + 1}}} \right)\), the answer should be “ 3*x^2-2*x/(x^2+1)^2 ”, which easily causes typing errors. It has been reported that many students experienced syntax issues in answering questions (Thaule, 2016). Several interfaces aiming to minimize math input difficulties have been proposed. The proposed interfaces include DragMath (Sangwin, 2012), a drag and drop equation editor in the form of a Java applet, which is used as one of the math input types for STACK. However, the interface requires Java and the input environment is restricted.

Most math e-assessment systems use CAS (e.g. Mathematica, Maple, Maxima etc.), which requests users to input mathematical expressions according to the rule of CAS. In order to improve the convenience of math input, template-based interfaces, such as DragMath mentioned above, are added to text-based interfaces. However, it is difficult for novice learners to adapt to current standard interfaces. For example, text-based interfaces accept input according to the CAS command syntax and it is difficult for the users to imagine the desired mathematical expressions because the input is not in the WYSIWYG format. However, structure-based interfaces have an advantage in that learners can operate in the WYSIWYG format. Moreover, they can input using math template icons. Therefore, they do not need to remember CAS command syntax in the case of the text-based interface. However, learners should understand the structure of the required mathematical expressions and should be able to select the math template icons in the correct order (Pollanen, Wisniewski, & Yu, 2007). Furthermore, it is cumbersome to make corrections later (Smithies, Novins, & Arvo, 2001).

In this section, we present two math input interfaces, MathTOUCH and FlickMath, developed by the authors. MathTOUCH accepts math input in the form of colloquial-style mathematical text and FlickMath supports math input on mobile devices.

3.1 MathTOUCH

MathTOUCH is a mathematical input interface with Java that facilitates conversion from colloquial-style mathematical text (Fukui, 2012). With this interface, users do not need to enter symbols that are not printed. For example, if users would like to enter \(\frac{{x^{2} + 1}}{3}\), they have only to enter “ x2+1/3 ”. They do not need to input parentheses for the delimiters, and a power sign (e.g. a caret symbol) as a list of candidates for each mathematical element is shown in WYSIWYG based on the user input (see Fig. 8.1). After all the elements are chosen interactively, the desired mathematical expression can be created.

Fig. 8.1
figure 1

MathTOUCH input procedure

Full size image

In a previous study, Shirai and Fukui implemented MathTOUCH in STACK and conducted two experiments—a performance survey (Shirai, Nakamura, & Fukui, 2015) and an eight-week learning experiment (Shirai & Fukui, 2014)—to evaluate the efficacy of MathTOUCH. The results obtained indicated that MathTOUCH enables tasks to be completed approximately 1.2–1.6 times faster than standard input interfaces (such as text-based interfaces and structure-based interfaces). Moreover, MathTOUCH was shown to have a high level of satisfaction with respect to math input usability. The results of the eight-week learning experiment show that students could practice using MathTOUCH on STACK at the same learning rate as with the standard input interface on STACK.

In 2016, Shirai and Fukui reconstructed MathTOUCH using JavaScript to make MathTOUCH available not only on Java-compliant devices but also on various other devices, and conducted a five-week learning experiment to evaluate the stability of reconstructed MathTOUCH. The results showed that students can study using reconstructed MathTOUCH on STACK as effectively as with the previous Java-version of MathTOUCH. The details including the data are available in Shirai and Fukui (2017).

3.2 MathDox and FlickMath

Nakamura, Inagaki, and Nakahara (2014a) developed an input interface for STACK using MathDox formula editor. MathDox formula editor facilitates the input of mathematical formulas two-dimensionally using a keyboard, and it also has a palette for input assistance (Fig. 8.2). Maple T.A.’s Equation Editor also realises the same type of mathematical expressions. However, with both editors, users have to switch keyboard between letters and numbers/symbols, especially when using mobile devices. Further, the editors do not reduce the complexity of the math input process for such devices.

Fig. 8.2
figure 2

MathDox input type for STACK

Full size image

Based on the MathDox input type, a new input type, FlickMath, was developed for using STACK on mobile devices (Nakamura & Nakahara, 2016). FlickMath allows the input of mathematical expressions by the flick operation (Fig. 8.3). The flick operation is carried out by placing a finger on the prepared keyboard, shifting the finger vertically or laterally, and subsequently releasing it. Japanese students often use flick input to input characters on their smartphones; therefore, inputting math using the flick operation should be natural for them. On tablet devices, a full keyboard is displayed and the flick operation is implemented. We conducted a survey on the usability and satisfaction levels of students. Our results, based on the average responses of 29 students, indicate that usability and satisfaction levels are higher when the flick input method is used to enter mathematical expressions as compared to the direct input method (Nakamura & Nakahara, 2017). As the input operation using FlickMath is less cumbersome than using the direct method on a smartphone, it could be effective for drill practices on mobile devices when students have spare time.

Fig. 8.3
figure 3

FlickMath input type for STACK

Full size image

4 Sharing Questions

When e-learning is employed, not only in mathematical science but also other subjects, content has to be prepared. However, math e-learning, especially math online testing, has a relatively short history and an individual teacher may not have sufficient content. As stated in Sect. 8.2.4, developing questions that can assess students’ answers and return suitable feedback appropriately is time-consuming. Accordingly, a question-authoring tool that reduces the amount of work involved was developed for STACK (Nakamura, Ohmata, & Nakahara, 2012). It is helpful for authors; however, in order to promote math e-learning effectively, it is perhaps more important to prepare as much high-quality content as possible and to share such content among teachers.

In this section, as effective ways to accumulate content, we discuss several content-sharing systems, including content-converting methods and our project that aims to share content among heterogeneous math e-learning systems.

4.1 Item Bank System: MathBank

In order to effectively promote student learning, well-structured questions must be shared, thereby allowing any registered user to access them. This helps to broaden the applications of e-learning to math and science subjects. Accordingly, we developed an item bank system (Nakamura, Taniguchi, & Nakahara, 2014b), called ‘MathBank’, open to the public at Mathbank.jp, https://mathbank.jp/.

We developed MathBank as a Moodle system wherein any user can register questions and search for registered questions after user authentication. When users register a question in MathBank, they are prompted to include metadata associated with the question such as grade, difficulty level, publicity level, and keywords. Questions can be registered by uploading an XML file via the interface. Users can also create questions on MathBank itself. After searching through the list of questions, users can download STACK questions in XML format and import the file to their servers for subsequent use. MathBank also provides time for registered questions to be tested straight from MathBank, which creates stored logs. The log is used to reconsider the difficulty level and to improve the quality of the questions in the system. MathBank was opened to the public approximately three years ago and, as of December 2016, 46 users were registered and approximately 200 questions were available on the system.

4.2 Maple T.A. Cloud

Tens of thousands of questions are available in the Maple T.A. Cloud (Maplesoft, 2013), a worldwide content-sharing system based on Maple T.A., ready for immediate use or customization. Content is available for a variety of subjects, including calculus, pre-calculus, algebra, differential equations, linear algebra, physics, chemistry, engineering, statistics, and economics.

4.3 Converting Content Between Different Systems and Building Common Base for Content Creation

The aforementioned question-sharing systems, Mathbank and Maple T.A. Cloud, are designed for specific math e-learning systems STACK and Maple T.A., respectively. One can share questions in each system only. A solution for increasing the amount of content is converting high-quality content created in one system to another system. There are some projects to convert questions among different math e-learning systems; e.g. conversion system between Maple T.A. and SOWISO (Baumgarten et al., 2015) and Maple T.A. to STACK conversion (Higuchi, 2016). However, conversion is not always perfect since some features of a system are sometimes not supported by the other system.

4.4 Necessity of Common Base for Sharing Content

We reviewed two content-sharing systems and some content-converting methods. However, it is certainly preferable to have a common base of shared content to accumulate content. An aggregation of content can be compared with an emerging “system” of interactive elements. For example, knowledge map (Chen, Chu, Chen, & Su, 2016; Reed, 2013) is a visualization of connections among subjects and shows an interaction between elements, which is similar to content-sharing systems. Further, the elements of content-sharing systems should be increased in number and, in other words, content-sharing systems should emerge. It is said that “evolutionary algorithms have a common base with evolution, since they are based on fundamentals of natural selection” (Davendra, 2010). Our project aims to share content among heterogeneous math e-learning systems based on a common base, i.e. mathematics e-learning question specification (MeLQS), which we believe realizes the objective of this work, i.e. to determine effective ways to carry out mathematics e-learning using CAA.

5 Mathematics E-Learning Questions Specification: MeLQS

In order to build a common base for sharing questions, we verified that the structures of the question data in STACK and WASM (see Sect. 8.2.1) are analogous (Yoshitomi & Kawazoe, 2013). In WASM, the data consists of question sentences with formatting similar to HTML that permits MathJax description, which has a similarly formatted answer format; Mathematica to analyze the input; a set of program (maybe including random functions) parameters that are randomly selected when actually used; feedback messages coupled with the return code of the analysis program above. It is well known that STACK has a similar constitution; conceptually, both the systems have the same structure. We attempted to convert the question data between the systems manually (Yoshitomi, 2014). First, we converted each piece of data to better understand text describing what the author wants to do and how the parameters are determined. We called the prototype of specification of the question data MeLCS, Mathematics e-Learning Contents Specification, (Yoshitomi, 2014), but have since changed the name to MeLQS, as defined previously.

We started the MeLQS project, a four-year project, with grant support in 2016, and aim to share the e-learning/e-assessment content in the universal format MeLQS. This format is expected to be easily exported to any format available in the world, including MATH ON WEB, STACK, and Maple T.A. Therefore, we use MeLQS as a common base of shared content to accumulate questions for math e-learning. After the preliminary analysis of structures of the questions of MATH ON WEB and STACK, we determined it appropriate to categorize the structures of questions as follows: question text and routine to create it; definition of answer column and answer type; routine to evaluate student answer and feedback. MeLQS also has metadata of questions: question name, subject, intention behind a question etc.

One of the most important features of MeLQS is that it is constructed with two specifications: concept design and implementation specification. In the following subsection, we describe MeLQS in detail.

5.1 Concept Design for Questions

Concept design is a specification of questions that describes how the question is designed according to concept, and it is described by mathematical statements rather than programming statements so that all mathematics teachers can understand the concept addressed by the question. The concept design is stored in a database and databased concept design can be viewed on the MeLQS web system or can be exported to TeX and PDF format. Therefore, concept design is useful not only for online tests but also paper-based tests. We implemented an authoring tool of concept design as a Moodle plug-in that allows users to, create a concept design, including metadata: question name, subject, intention behind a question etc., step-by-step as shown in Fig. 8.4. At the present stage, how to input mathematical expressions is not fixed, but TeX is preferable for the preview. We also plan to familiarize the editorial function to all teachers with support from MathTOUCH (see Sect. 8.3.1).

Fig. 8.4
figure 4

Authoring tool of concept design of MeLQS

Full size image

5.2 Implementation Specification for Questions

The standard of implementation specification is being formulated as of May 2017. It is considered that those who have experience in authoring questions for online tests would create implementation specification based on the suggested concept design. In the implementation specification, details of settings of questions defined as dependencies on each math e-learning system are eliminated. For example, input of mathematical expression should not be dependent on each CAS syntax. Authoring tool like Fig. 8.4 will be developed in the future.

5.3 Implementation of Questions in Math E-Learning System

Questions based on the implementation specification can be used in each mathematics e-learning system. We plan to provide MeLQS as a cloud service with functions that enable users to author the question data and to export and import them to heterogeneous systems. At the present stage, the evaluation procedure implemented in STACK and MATH ON WEB cannot be reflected to Maple T.A. but the implementation is being considered by referring MeLQS.

In the future, we aim to make it easy for virtually all teachers to participate in the service. All the users can use this system freely but are expected to provide feedback about the effects or issues associated with the downloaded teaching materials to the community. We expect that significant use of the service will promote more practical usage and increase the efficacy of math e-learning systems.

6 Conclusion

We briefly reviewed the three main math e-learning systems used in Japan and outlined two problems associated with using them: inconvenience in inputting answers and heavy content-development workload. In order to solve the problem of inconvenience in inputting mathematical expressions, we developed two math input interfaces: MathTOUCH and FlickMath. Currently, both the math input interfaces are implemented only in STACK, but as they are HTML5-based, they can be applied to other systems. Sharing content reduces the problem of heavy content-development workload. Maple T.A. Cloud and MathBank are used to share mathematical questions by Maple T.A. and STACK, respectively. Although similar types of mathematical questions are present in both the systems, these questions cannot always be interchanged between them. Many questions have also accumulated in MATH ON WEB, but these questions are not compatible with other systems.

Collecting well-constructed mathematical questions in the promotion of math e-learning is undoubtedly important. In order to share questions among heterogeneous math e-learning systems—MATH ON WEB, STACK, and Maple T.A.—we have started a four-year project wherein the first step is to design the universal format MeLQS. We believe MeLQS, as a common base, is necessary for contents and usage of e-Learning systems in undergraduate mathematics education to increase in number. This is our answer to the research question in the present paper: what is necessary to realize effective use of mathematics e-learning using CAA. Eventually, we plan to provide MeLQS cloud service with functions that enable users to author question data and to export and import them to heterogeneous systems. We believe that heavy use of the service will promote more practical usage and increase the efficacy of math e-learning systems.