Keywords

1 Introduction

The Implementation Opinions on the Construction of First-class Undergraduate Curriculum (China Ministry of Education Higher Education Document No. 8 [2019]) pointed out that the curriculum system is the foundation, core and key to the quality assurance of talent cultivation [1] and that the construction of golden courses of “one degree in both genders” standard is inseparable from high-quality assignments. However, the fact is that homework in Chinese universities is generally not valued by teachers and students due to its weakness and inhibition, and lacks careful design, which has become the “short board of the curriculum” [2]. This is reflected in the low frequency of updating the homework question library, the high degree of similarity with online questions, and the single difficulty of the questions, which leads to students being keen to search for questions on the Internet, copy others’ homework, and complete homework tests quickly and at the lowest cost, resulting in a serious decline in learning initiative.

In order to solve the aforementioned problems of collegiate assignments, this paper proposes a semi-automatic fuzzy test-based coursework framework to provide students with personalised questions. After practical analysis, it can effectively enhance students’ learning initiative by preventing similar questions, dynamically adjusting the difficulty level and allowing students to repeatedly self-test, and providing teachers with scientific data on the real learning situation.

2 Related Work

2.1 Innovation in Homework Questions

According to Bloom’s cognitive model, knowledge can be divided into six types: remembering, understanding, applying, analysing, evaluating and creating [3]. Tan Xiaoxi [4] suggested that assignments are a tool that can connect teachers with students and students with each other in a dialogue, so researchers also think of ways to ensure that assignments are unique and expected to collect learning accurately so that remedial measures can be made in a timely manner to effectively enhance teaching and learning.

Since COVID-19 in 2020, more and more scholars have been innovating and making breakthroughs in the way homework is assigned and corrected. Li Jianzhang et al. [5] proposed to split large computational questions into multiple blanks and then use the pre-prepared conditions and answers in the database to customise the questions for students; Huang Hengjun [6] et al. used a housework system based on resampling techniques to achieve differentiated assignments at low cost and achieved good implementation results, but it was mainly used for data analysis courses as well as experimental data. It can be seen that the current functional requirements of university coursework systems are mainly focused on the ability to randomly vary, to repeatedly self-test, and to automatically correct, etc. Fuzzy testing techniques meet the needs of these precisely.

2.2 Fuzzy Test

Fuzzy testing is an automated software testing technique that is centred on the ability to automatically or semi-automatically provide random test cases as input to a target system that can be iteratively tested to expose the possible software vulnerabilities. Fuzzing refers to the automatic generation and execution of tests, and the random data input to a fuzz test is often referred to as ‘Fuzz’.

Yang et al. [7] spent three years reporting over 325 vulnerabilities to developers of compiler community. 2019 Google Project Zero [8] reported the number of vulnerabilities found through fuzz testing, with a percentage of 37%, and fuzz testing has now become an important technique for detecting vulnerabilities in systems.

The basic workflow of fuzzy testing can be divided into 5 steps: Preprocessing, Input building, Input selection, Evaluation and Post-fuzzing. The first and fifth parts are the preparation work before the fuzzy test starts and the closing work after the fuzzy test is finished, while the actual test parts are the second to fourth parts.

3 Questions Design

3.1 Preprocessing

According to Bloom’s cognitive model, in the core course Principles of Computer Composition for computer science, preprocessing adjusts the format of questions through human intervention to meet the requirements of automatic computer review and to limit the possibilities of student answers as much as possible.

For example, traditional calculation questions can be designed as step-by-step fill-in-the-blank questions, leaving blank content on each key step of the solution for students to fill in, or the calculation questions can be transformed into programming questions [9]. After this session each question E can be considered as a combination of items I, represented by the Eq. (1):

$$ E = \left. {\left\{ {I_1 } \right., \ldots ,I_n } \right\}^T \left( {n \geqslant 1} \right) $$
(1)

where I consists of the sets, including templates T, variables V and relations S, as in the Eq. (2):

$$ I = T + V + S(V,A) $$
(2)

S(V, A) represents the mapping relationship from the set of variables V to the set of answers A, i.e. each variable element v can be calculated in a fixed way to arrive at the corresponding answer a. It is noted that V(S) is the definition domain of the calculation relationship S, and A(S) is the value domain of the calculation relationship S. Therefore, when using the same template T, the larger the number of variables set V, the smaller the chance of identical answers, which can effectively reduce the possibility of plagiarism by students.

3.2 Variable Construction

The variable construction focuses on the random variation of input data in the correct format of the original question. The analysis of the characteristics of question types determines the types of importable variables, the allocation strategy and priority of the variables, and finally constructs some input data as initial test cases based on this information.

  1. 1)

    Types of judgement, choice questions

Judgement type Ejdg is formed by joining multiple terms In(n ≥ 1) by a Cartesian Product, noted as:

$$ E_{jdg} = Product\left\{ {\left. {I_1 , \ldots ,I_n } \right\}} \right.\left( {n \geqslant 1} \right) $$
(3)

By randomly generating Boolean variables V = {Yes, No}, A = {True, False}, it follows that I consists of a pair of positive and negative propositions, i.e. I = {P, ¬P}.

If \(\left| {\text{N}} \right|\) denotes the number of elements in a set, then the equation |Ejdg| = 2n indicates that there are 2n different combinations. For example, when n = 2, |Ejdg| = |I1 × I2| = |{X, ¬X} × {Y, ¬Y}| = 4 expresses that students can make a judgement from 4 different descriptions of the question (Fig. 1).

Fig. 1.
figure 1

Three-Floor random structure

Full size image

Single and multiple choice questions are essentially the same in that they are made up of a number of judgement questions (3). The difference between the two lies in the number of combinations of options. Single-choice questions consist of the only correct or the only incorrect combination, whereas multiple-choice questions consist of two or more correct or incorrect combinations, which is expressed as:

$$ E_{sch} = Select\;in\left\{ {\left. {E_{jdg1} , \ldots ,E_{jdgn} } \right\}} \right.\,{{\& }}\,\left| {T\;or\;F} \right| = 1 $$
(4)
$$ E_{{\text{m}}ch} = Select\;in\left\{ {\left. {E_{jdg1} , \ldots ,E_{jdgn} } \right\}} \right.\,{{\& }}\,\left| {T\;or\;F} \right| \geqslant 2 $$
(5)

Thus, whether it is a judgement, single or multiple choice question, multiple combinations of similar but different answers can be expressed through the construction of Boolean variables V, which are randomly assigned to students for completion as homework in the course. In addition, there is also a category of multiple choice questions which are answered by filling in numbers, which will be discussed along with the following.

  1. 2)

    Types of blank, calculation questions

Assuming that a computational relation S(V, A) exists in these questions, then in order to find a ∈ A(S), we need to understand the definition description in T and then construct the computational model S based on the data condition v ∈ V(S) given in the question, noted as:

$$ \exists S(V,A),S:V|T \to A $$
(6)

In this case, the computational relationship S is the focus of the question, and the application of students’ knowledge is repeatedly reinforced by constructing different data conditions v.

Questions allow for constructing one-dimensional vectors {V1, V2, …, Vn}T to meet the demand of combinations of computational relationships. So students can randomly compute one of {A1, A2, …, An}T and teachers can analyse the level of detail of students’ knowledge. It is worth noting that there is a progressive or decentralised relationship between the computational relationships, depending on the need for a knowledge quiz.

  1. 3)

    Types of program cloze questions

The template T for this type of question is changed to a design diagram or procedure in which V = {b1, …, bn} are pre-designed in T. A number of blanks are randomly selected from V to form the set named B, while the set of other remaining blanks called V-B, is directly replaced by the corresponding answer AV-B, noted a

$$ S:B|(T + A_{V - B} ) \to A_B ,T = \left\{ {\left. {Picture,Program} \right\}} \right. $$
(7)

3.3 Variable Filtering

The input selection filters the constructed data and sets constraint policies. It tries to filter out invalid input data in advance, including conflicts such as data types, data ranges, repetitive or contradictory descriptions, and undefined behaviour of the compiler. And it guarantees the quality of the input data and the accuracy of the topic description.

There are various strategies for setting constraints, e.g. input data for some topics can be determined by comparing the results of multiple generic compiler runs, with all runs agreeing before passing; e.g. teachers can directly delineate the types of data that can be input and the range of data or teachers collect feedback from students on the vulnerabilities of assignment problems through the assignment platform and fix them.

3.4 Difficulty Evaluation

This part as the final step in the fuzzy test process, will select an appropriate assessment indicator for optimising a more rational strategy and reflecting students’ true level of performance on knowledge. At this stage, current research has focused on the performance of the assignment on two indicators: the failure rate of the questions and the difficulty factor.

According to Bernoulli’s law of large numbers, let nA be the number of times event A occurs in n independent repetitions. p is the probability of event A occurring in each trial, then for any positive number ε > 0, we have

$$ \mathop {lim}\limits_{n \to \infty } P\left\{ {\left. {\left| {\frac{n_A }{n} - P} \right| < \varepsilon } \right\}} \right. = 1 $$
(8)

Bernoulli’s law of large numbers states that the frequency of an event nA/n converges to the probability of the event p. This means that when n is sufficiently large, there is little chance that the frequency of the event will deviate significantly from the probability. From the principle of practical extrapolation, the frequency of an event can be used in place of the probability of an event in practical applications where the number of trials is large.

Since the difficulty factor for each question is proportional to the probability of missing a mark, the difficulty factor can be obtained from the probability of missing a mark or the rate of missing a mark. Each question has a different probability of losing marks depending on the variables. If a mark scheme is used, the mark is lost if the answer is correct, and the mark loss rate for a particular assessment point is defined as the difficulty factor, which is given as:

$$ P_{final} = P_{base} \ast \left( {1 + P_{offset} } \right) = \frac{N^e }{N} \ast \left( {1 + \frac{N_v^e }{{N_v }}} \right) $$
(9)

where \(N_{\text{v}}^e\) and \({\text{N}}_{\text{v}}\) denote the cumulative number of missed scores and hits respectively for a question after adding a random variable (offset), \(N^e\) and \(N\) denote the cumulative number of missed scores and hits respectively for a question using the baseline condition, \({\text{p}}_{base}\) and \({\text{p}}_{{\text{offset}}}\) denote the rate of missed scores for a question using the baseline condition and the rate of missed scores for a question after adding a random variable (offset) respectively.

Sometimes a question often has more than one assessment point, so the difficulty factor of the question is defined as the average of the difficulty factors or probability of losing marks for all the assessment points of the question, i.e. the difficulty factor of a question with many assessment points, which is recorded as:

$$ P_{avg} = \frac{1}{m}\sum\nolimits_{i = 1}^m {P_i} $$
(10)

In the online assignment platform system, whenever a student submits an answer sheet, the system performs a self-study of the test results submitted by the candidate, i.e. recalculates the difficulty factor and modifies the difficulty factor of the corresponding test question to suit the actual solving ability of the current student. As the number of assignments and the number of assignments increases, the difficulty factor gradually approximates the actual situation.

4 Framework Design

The different types of questions consist of a variable V, a template T, and a mapping relation S(V, A). Obviously, homework questions can be generated in a program, where the variable <V> corresponds to the program input, the template <T> to the static content printed out by the program, the mapping relation <S> to the logical transformations in the program, and the answer <A> to the program output. In addition to this, the program incorporates a seed s for randomly generated variables and a constraint r. If the mapping relation S can be expressed by a simple formula, this can be achieved by using the Office/WPS Excel formula. The specific framework design for the homework is shown in Fig. 2.

Fig. 2.
figure 2

Framework for semi-automated homework based on fuzzy test

Full size image

The following are some details from Fig. 2:

  • The teacher selects a particular question as the Parent Question after a number of assignment questions have been randomly generated through the same program. All the initial difficulty factors are obtained from databases or teachers.

  • The system can be programmed to automatically judge answer sheets and update difficulty factor of each question.

  • The assignments will be adaptively assigned to students based on the difficulty of the question and their past performance with as high a coverage as possible through the intelligent systems.

5 Teaching Effectiveness

5.1 Teaching Data Comparison

The teaching class of the pilot course on Principles of Computer Composition was divided into two groups, the control group (161 students) and the experimental group (161 students) both used the online platform to do the exercise. The difference is that each question in the control group is identical, and each question in the experimental group has the same origin, but different variables and answers.

It can be seen from Table 1 that the experimental group, after reinforcement with anti-plagiarism and self-testing of randomly differentiated homework, had significantly higher learning initiative, more communication and discussion, deeper understanding of knowledge points and higher achievement of course objectives than the control group.

Table 1. Data Comparison of Teaching Activities
Full size table

5.2 Questionnaire Analysis

In order to investigate the effectiveness of the customised system of fuzzy test assignments from the perspective of the teaching class, regular questionnaires were administered to all students in both groups and the results showed that the majority of students had a positive opinion of the randomly selected question format.

In particular, a multiple-choice question that indicated a preference for the assignment was asked whether you would ask students directly “What is the answer to this question” or “What is the process for this question” when they did not know how to do the question, and over 90% of students of experimental group chose the latter.

6 Conclusion and Suggestion

In this paper, we use fuzzy test technology to build a course assignment platform, which automatically or semi-automatically generates random data into assignment templates by randomly varying the input data in the correct format of the original questions, and combining them into assignments with variant inputs for students to practice or take tests. The fuzzy mechanism for assignments both eliminates the majority of plagiarism by students and provides an efficient feedback mechanism for teachers and students, reinforcing the quality of teaching and learning, especially online.

But homework is still dependent on the core program that generates fuzzy data, and this requires a lot of programming effort on the part of the teacher. Therefore, future work will, consider lowering the threshold for using the coursework platform and studying the use of visualisation for the purpose of programming questions. These will continue to be studied in depth and continuously improved in subsequent work.