Keywords

18.1 Introduction

Well known effective educational model for less prepared learners is one-on-one tutoring [1]. But, one-on-one tutoring is not a realistic solution for many learners because of cost. This motivated us to develop Intelligent Tutoring Systems (ITSs) with one-on-one tutoring capability for calculus and linear algebra for college level learners. Even though ITSs are becoming popular among learners at pre-college level mathematics courses [4], designing ITS which accurately diagnose learners’ knowledge structure, skills, and styles is not easy. According to [7], to diagnose learners’ knowledge structure, the generated question should be short answer question but not multiple choice questions.

In this article, how educators’ knowledge structure map can be utilized to diagnose a learner’s knowledge structure is shown. Then how the knowledge score can be used to assess a learner’s understanding of the material is shown. Furthermore, how the relative distance is utilized for implementing an adaptive feedback system into JCALC is shown.

18.2 Assessing Learner’s Knowledge

18.2.1 Experienced Mathematics Educator’s Knowledge Structure

It is often said that experienced mathematics educators can often tell what causes him/her to make a mistake in the exam or what types of problems learners might fall into by grading exams or looking at what learners are writing. This forces us to study that how experienced mathematics educators can tell the cause of problems by reading a learner’s solution written on the paper. Here, ten experienced mathematics educators are chosen from the mathematics department of our school. Then they are given the following learner’s responses, and asked why these learners made mistakes.

  1. (1)

    A learner writes 2(x 2 + 3x)3(2x + 3) as to the question of “Find the derivative of (x 2 + 3x)4”.

  2. (2)

    A learner writes −sin(3x + 1) as to the question of “Find the derivative of cos(3x + 1)”.

  3. (3)

    A learner writes −2/(x + 1)2as the answer to the question of “Find the derivative of (x − 1)/(x + 1)”.

  4. (4)

    A learner writes xex + x + c as the answer to the question of “Evaluate ∫xexdx”.

  5. (5)

    A learner writes \( \det \left( {\begin{array}{*{20}c} { - 1} & 0 \\ 1 & 2 \\ \end{array} } \right) \) as the answer to the question of “Find the (1, 2) minor of\( \left( {\begin{array}{*{20}c} 1 & 3 & 2 \\ 2 & { - 1} & 0 \\ 0 & 1 & 2 \\ \end{array} } \right) \)”.

  6. (6)

    A learner writes \( \det \left( {\begin{array}{*{20}c} 2 & 0 \\ 0 & 2 \\ \end{array} } \right) \) as the answer to the question of “Find the (1, 2) cofactor of\( \left( {\begin{array}{*{20}c} 1 & 3 & 2 \\ 2 & { - 1} & 0 \\ 0 & 1 & 2 \\ \end{array} } \right) \)”.

  7. (7)

    A learner writes \( \left( {\begin{array}{*{20}c} 1 & 3 & 2 \\ 2 & { - 1} & 0 \\ 0 & 1 & 2 \\ \end{array} } \right) \) as the answer to the question of “Find the cofactor expansion of \( \det \left( {\begin{array}{*{20}c} { - 1} & 0 \\ 1 & 2 \\ \end{array} } \right) \) along the 1st row”.

The learners’ responses are collected and sorted and shown in Table 18.1.

Table 18.1 Typical responses by experienced mathematics educators
Full size table

The experienced mathematics educators’ responses can be explained by using a concept map. For example, consider the response (1). The experienced mathematics educators read the learner’s solution 2(x 2+3x)3(2x+3). Then they compared the learner’s solution to the right solution. To do so, they have differentiated the given function by themselves. In other words, they have to recall the chain rule and apply it correctly. Furthermore, they have to recall the differentiation of the power function and apply it correctly within a short period of time. With all these process, they have noticed that the derivative of power function is essentially in the right form. Furthermore, the chain rule is applied correctly. Therefore, the experienced mathematics educators’ response for the question (1) becomes like the one in Table 18.1.

Now notice that every experienced mathematics educator used the chain rule and the derivative of the power function. Thus, every experienced mathematics educators’ knowledge structure is very similar. Even though the knowledge of an individual expert consists of both a cognitive element—the individual’s viewpoints and beliefs, and a technical element—the individual’s context specific skills and abilities [2, 6], experienced mathematics educators’ knowledge structure can be used as the basic knowledge structure about how to solve problems in calculus and linear algebra.

18.2.2 Assessing Learner’s Knowledge by the Relative Distance

By defining the ratio or the difference of the evaluated values of a learner’s input and a generated correct answer, it is possible to assess a knowledge state of a learner. As in [10, 11], we first define the distance d: Let vin and vca be defined as follows:

$$ \begin{aligned} vin & \, = {\text{the value of the input evaluated at certain point}}. \\ vca & = {\text{the value of the correct answer evaluated at certain point}}. \\ \end{aligned} $$

Then define the distance d as follows:

$$ d = \left\{ \begin{gathered} {\text{the difference of }}vin{\text{ and }}vca \, \hfill \\ {\text{the ration of }}vin{\text{ and }}vca \hfill \\ \end{gathered} \right\} $$

For if a learner’s input value is far from the correct value, the distance d becomes large. This type of phenomena can occur if a learner does not know a material at all or some. In this case, even an experience educator cannot conclude whether the learner knows a material a little or none. Thus, it is necessary for alternative way to assess learner’s understandings.

Define rd by the following equation:

$$ rd = \frac{d}{\text{evaluated value of correct answer}} $$
(18.1)

If the value rd is large, then d must be very large compared with the evaluated value of the correct answer. Then it is quite natural to assume that the learner does not know much about the material. On the other hand, if the value rd is small, it is natural to assume that the learner knows the material a little. With this reason the value of rd is called the relative distance, now to assess a learner’s knowledge structure, the following example explains the usage of the relative distance. Suppose that the system generated question is given by “Differentiate 2(x 2 + 3x)4” and a learner’s input is 2(2x + 3)(x 2 + 3x)2. Furthermore, the system generated solution is 8(x 2 + 3x)2(2x + 3). Then the evaluated value of the learner’s solution at x = 1.315 is equal to 2057.11, and the evaluated value of the system generated solution at x = 1.315 is 8228.45. Since the evaluated values of these two expressions are not equal to each other. Thus, it is possible to tell the learner’s solution is wrong. Now, calculate the relative distance defined above. Then

$$ rd = \frac{{\left[ {8(x^{2} + 3x)^{3} (2x + 3)|_{1.315} - 2(2x + 3)(x^{2} + 3x)^{3} |_{1.315} } \right]}}{{8(x^{2} + 3x)^{3} (2x + 3)|_{x = 1.315} }} = \frac{3}{4} $$
(18.2)

Here rd is given by the simple fraction 3/4. Note that by the Sect. 18.2.1, the experienced mathematics educators assess the learner’s knowledge structure by checking each step necessary to obtain the right solution. This tells us that the learner’s knowledge structure can be assessed by checking the relative distance. We also note that the experienced mathematics educators concluded that the learner probably made the simple mistake. Therefore, the relative distance is given by the simple fraction implies that the learner’s knows the material, but made a careless mistake.

18.2.3 Assessing Learner’s Knowledge Structure by the Experienced Mathematics Educator’s Knowledge Map

To assess a learner’s knowledge structure, one well known method is concept mapping. According to [7], to construct a good concept map, it is important to begin with a domain of knowledge structure that is very familiar to the person constructing the map. Following this suggestion, we first made sure that the learning outcomes of the subject such as calculus and linear algebra usually taught in college mathematics. Then the performance criteria for each concept for which learners are expected to learn is created. An example of the differentiation is shown in the following Table 18.2.

Table 18.2 Performance criteria
Full size table

In the process of deciding what to expect for students, we learned that a relational is helpful. Note also that superior performance is dependent not only on domain knowledge but also on intimate familiarity with the relational structure of domain objects in a problem situation [5, 8]. This suggested that it is necessary for our mathematics software to be equipped with not only concept maps but also relational structures.

Now to check to see whether the learners understand the concept, questions containing necessary knowledge must be created and tested. This suggests that questions generated by our system must be split into finer questions which are more familiar to the learner. Then by examining how well learners answer to the finer short-answer questions, a concept map for each learner can be created. Note that in the analysis for the use of abstract concepts, a larger number of links were expected to be attached to abstract concepts in the high performer network than in the low network. Now using the knowledge structure of experienced mathematics educators, it is possible to identify the key concepts that apply to this domain. Thus, the concept map of experienced mathematics educators’ knowledge structures is used to refine a short-answer question.

For example, the concept map of the differentiation of composite function is shown in the following Fig. 18.1.

Fig. 18.1
figure 1

The concept maps for derivative of composite functions

Full size image

Experts characteristically use more abstract concepts to solve a problem than novices [9]. Since experts chunk or group their knowledge differently, their mental models should be characterized by groupings around abstract concepts.

Now to implement the concept map into JCALC, we introduce the “knowledge score”. The marks such as 0.2 and 0.3 on arrows are called “knowledge score” which indicate the basic knowledge needed to obtain a correct concept. Note that the knowledge scores on top row adds up to 1, and the knowledge score added vertically adds up to the one of top scores. In other words, to be able to differentiate a composite function, the knowledge about composite function consists of 20 %, the knowledge about the chain rule consists of 50 %, and the knowledge about the basic rules of differentiation consists of 30 %. These percentages are derived by adding the necessary knowledge needed to acquire before completing the top row knowledge.

The explained knowledge structure map is shown in the following Fig. 18.2.

Fig. 18.2
figure 2

Educators’ knowledge structure map

Full size image

18.3 Adaptive Tutoring System JCALC

18.3.1 How to Tell a Right Answer from a Wrong Answer

Look at the question of finding the integral of the function “1/1 + sin x”. Then a learner’s input such as “tan x − sec x + c” is a right answer. But another learner’s input “\( {{2\tan \frac{x}{2}} \mathord{\left/ {\vphantom {{2\tan \frac{x}{2}} {\left( {1 + \tan \frac{x}{2}} \right) + c}}} \right. \kern-\nulldelimiterspace} {\left( {1 + \tan \frac{x}{2}} \right) + c}} \)” is also a right answer. Thus to judge a learner's input is a right answer or not, it is not possible to list the right answers and compare words by words. For this reason, it should be noted that any system which produces multiple choice questions and answers is not suited for college level mathematics. This suggests that to develop an adaptive system for college level mathematics, any system should be able to handle short-answer questions. Furthermore, according to [4], short-answer questions have the advantage of avoiding cueing rather than selecting or guessing from options supplied. Thus, in our system, a learner must enter his/her answer in the form of mathematical expressions. This also suggests that our system must be able to read a learner’s input and be able to tell whether it is a right answer or not.

The expressions for right answers are not unique, rather unlimited. Thus, preparing all expressions for right answers in database, and checking whether learners’ answer is in database to decide the learners answer is correct is not plausible. So, developing some other way to tell right answers form wrong answers is expected.

Look at the values of the power of x, say at x = 1.3. Then the values of the power of x are given as follows: x 2 = 1.69, x 3 = 2.197, x 4 = 2.8561,…. Now note that the last digit’s decimal place increases one place each time. This means that the counting the decimal places appear in expression, it is possible to tell which power of x the expression contains. Which in turn implies that two polynomial expressions are evaluated to be equal imply that they are exactly the same expressions. So to decide the learner input is a right answer or not, it is only necessary to evaluate the learner’s input and the system generated solution at some point. Noting that every elementary function can be approximated by the polynomial, we can determine that a learner input is right answer or not by checking the value of a learner input and a correct answer at some point. More detailed discussion is given in [10].

18.3.2 Inferring Learner Knowledge Structure

It was shown in the Sect. 18.3.1 that it is possible to determine the learner input is right or not by evaluating the correct answer generated by JCALC and a learner input at certain value. We studied this method carefully to notice that when the value of the correct answer and the learner input are different, their difference or ratio has some tendency among group of learners. Suppose that a displayed question is “find a derivative of (x 2 + 2x + 3)4” and a learner’s input is “(x 2 + 2x + 3)3(2x + 3)”. Furthermore, the correct answer generated by JCALC is “4(x 2 + 2x + 3)3(2x + 3)”. Looking at the learner’s input, anyone with calculus teaching experience judges that the learner has the knowledge of derivative of composite function because he/she has took care of derivative of power function then worked inside function.

Suppose this time that the learner input is “4(x 2 + 2)2”. Then again anyone with calculus teaching experience would say that this learner did not master the rule of derivative of composite functions. It is because the derivative of the inside function is taken before the derivative of the power function. This time it is not easy to design our system to judge the same way as the experienced mathematics educator. For learners inputs vary many ways and it is impossible to cover all.

Now as explained in Sect. 18.2.3, the knowledge score for each performance criterion is calculated. Then by adding the knowledge scores to the learner’s knowledge structure, the complete knowledge structure of the learner can be obtained. Thus to infer a learner’s knowledge structure, adding the knowledge scores for each question is only thing to do.

18.3.3 Feedback

It is noted in [3] that any educational software needs to give a quick feedback to encourage a learner to study more. This suggested that our system ought to have a few types of feedbacks. For example, using the relative distance explained in Sect. 18.2.2, the simple fraction rd can give the feedback like “Careless mistakes. Try again”. The small value of the relative distance can give the feedback like “Close to the right answer. Try again”. One more feedback needed to our system is the statement such as “Expression has not been simplified”. One way to accomplish this is to compare the number of terms in the generated solution and the learner’s solution. After a learner’s input are read, interpreted, and the relative distance and the knowledge score are calculated, three different types of hints will be displayed. Since the knowledge score is supposed to assess a learner’s knowledge structure, the sum of knowledge scores gives more valuable information about how much learner knows.

Now to show how adaptive hints are generated and displayed, all subjects and performance criteria of calculus and linear algebra are checked. Then short-answer questions are divided into 44 groups depending on the number of steps used to solve these questions. For example, when to generate a question of differentiating a product of functions, a collection of techniques which gives a product rule is searched. Then using the function generated, a hint explaining what to do to solve this question is displayed. If a knowledge score is less than 1, then using the displayed question, hint for which experienced mathematics educator might give will be displayed. If learner cannot get a right answer, another hint will be displayed.

18.4 Conclusion and Future Work

We implemented the assessing method explained above into JCALC and tested the assessing method is valid or not. To verify whether the hypothesis is true or not, we use the hypothesis test for slope of regression line. The null hypothesis is the slope = 0. Then we obtain the following results: the standard error SE is given by SE = 0.253, T-score is given by T = 1.926. From these, we obtain that P(t > 0.1926) = 0.0415. This shows that the p-value is less than the significant level (0.05), and we cannot accept the null hypothesis.

We are currently running our system JCALC on web and collecting learners’ data as much as possible. The data collected contains the following information: learner’s ID, subject selected, section selected, and number of questions tried, number of right answer, time spent for solving each question, expression inputted by learner, which type of feedback is shown, system generated solution. From the information above, each generated question is rated for difficulties. Then using this rating, the future system should be able to provide questions which emphasize to fill the learner’s week point.