Keywords

1 Introduction

Presenting Introductory Computing Courses (ICCs) to a large number of students requires educators to utilise effective educational practices in today’s modern classrooms [6–8]. Tertiary institutions presently are experiencing large enrolment numbers for ICC. An increase in the number of students also means a linear increase in the number of assignments and tests that need to be graded [9].

Introductory Computing Courses, also referred to as Computer Fundamentals or End User Computing, present their own challenges with students from different backgrounds and a vast difference in skill levels, from complete novices to experienced students. Student content retention can be positively reinforced by increasing the number of exercises, problems and assignments completed by the students in an ICC [9]. However, increased assignments result in increased workloads for instructors as the amount of work to grade assignments increases [6].

E-assessment and the use of Automated Grading Systems (AGSs) can assist with the grading and feedback provided to students. These systems are generally researched from the perspective of instructors, e-learning experts and educational technologists, however there is limited research that focuses on the students’ perception of e-assessment [5]. The Department of Computing Sciences at the Nelson Mandela Metropolitan University (NMMU) is exploring different options with regards to the submission and marking of MS-Office documents as part of the ICC, called the Computing Fundamentals Module (CFM). The need was identified to determine the best method of submission from a student’s perspective.

The research problem investigated in this research study was that educators are not aware of the issues that need to be addressed when considering an assignment submission method, from a student’s perspective. The main research objective of this study was to compare three different submission procedures for submitting MS-Office assignments in the CFM. The focus was thus on student perceptions of the method used to submit assignments assessing their MS-Office skills and to provide feedback on the effectiveness and efficiency of the different methods.

The research context and research methodology for the study is presented (Sect. 2) and followed by the discussion of the research results (Sect. 3). The paper concludes with findings, recommendations and future work (Sect. 4).

2 Research Study

The research was carried out with students enrolled in the Computing Fundamentals Module (CFM) presented by the Department of Computing Sciences at NMMU in February 2016. This section first describes the current method of practical submission (Sect. 2.1). This was one of the three methods included in the study (Sect. 2.3). The research methods used to conduct the study are explained (Sect. 2.2) as a prelude to the results presentation (Sect. 3).

2.1 Current Situation

Students enrolled in the CFM are required to submit weekly practical assignments requiring students to format and/or adapt a document, spreadsheet or powerpoint file based on a set of instructions. Students need to submit the assignment file by saving it in a special network folder for the purpose of module submissions.

There are over 1000 students enrolled in the module, with the result being that the task of marking the assignments that may be required for submission weekly, is impossible for lecturers. Students receive assistance and feedback from student assistants in practical sessions or if they approach the lecturers. However, they do not receive any feedback about the accuracy of their documents after submission. Feedback on whether or not the method of saving documents in the submission folder is preferred by students, would be useful.

2.2 Research Methods

The research study used the survey research approach. The use of surveys incorporating Likert scale questions, attitudes and feelings can be quantified [5] in order to make generalisations and inform decision making [4].

The research aimed to compare three different systems. A survey specifically designed for the evaluation of system usability is the System Usability Scale (SUS). SUS enables a researcher to get a measure of the perceived usability of a system [1]. The SUS questionnaire consists of 10 Likert scale items or statements related to system usability in terms of effectiveness, efficiency and satisfaction [2]. In this study, the 5-point Likert Scale was used, where one was Strongly Disagree and five Strongly Agree.

It was decided to use an assignment that was due for submission. Participation was voluntary. In total, 45 students agreed to participate in the study. The participants were required to submit the assignment using the first method, then complete the SUS survey for that method before moving to the next method and doing the same. Once all three methods were completed the participants were also asked to directly compare methods with each with regards to the following statements:

  1. 1.

    More likely to use in future,

  2. 2.

    Easier to use,

  3. 3.

    Learn more quickly,

  4. 4.

    More confident using,

  5. 5.

    Allows to work efficiently,

  6. 6.

    Allows to work effectively,

  7. 7.

    Marks more accurately,

  8. 8.

    More confident that submitted,

  9. 9.

    More satisfied that submitted, and

  10. 10.

    Provides better marking feedback.

For each of the statements, participants were asked to choose between Method 1 and Method 2, Method 2 and Method 3, and Method 1 and Method 3. Participants could also indicate no difference. The order of the methods were changed to avoid bias in the results due to learnability.

2.3 Methods of Submission

Three methods of submission and grading were evaluated in this study, namely the submissions folder method, the use of Moodle, and an online system. Essentially the only difference between each was the method of submission. All three used the same AGS to grade the assignments and provide participants with a mark report.

The submissions folder method was the method currently used by students. Each student is allocated a folder on the network for assignment submissions. They simply have to save their assignment files in this folder for submission purposes. For the purposes of the study, participants were required to indicate once this task had been completed so that the submitted file could be graded immediately. Once the marking process was completed an email was sent to the participant with the mark report as an attachment.

The Moodle submission method required participants to sign into the NMMU Learn site (a Moodle learning site) and submit their assignment. Participants were familiar with using the Moodle site for module quizzes and to download module information. Participants were provided with instructions on how to upload and submit their assignment files. Participants had to indicate once the file was submitted so that the marking process could be initiated manually. A mark report was emailed to the participant as an attachment.

The online submission method required participants to navigate to an online site where they could upload the file they wished to submit. The online system was able to check that files are named correctly, informing participants if the file name was not correct. The online system initiates the marking of the assignment file and, once complete, indicates to participants that the file was successfully submitted and provides a link to download the mark report.

The same marking system was used to grade the assignments submitted in all three methods. Originally the study planned to also evaluate the marking system by comparing the new system to the system used previously, namely the SAM assessment system [3]. However, the timeline for the study was delayed and SAM was no longer licensed for use at the time of the surveys.

3 Data Analysis and Results

The SUS surveys for the three methods were analysed by looking at the overall mean score for each method as well as considering the mean response for each statement. Table 1 indicates that the submission folder method obtained the highest SUS score. Both the submission folder method and the online system method scores were above average, where average is a score of 68 [2]. The Moodle submission method scored just below average.

Table 1. Mean SUS scores for each method (\(n=45\))
Full size table

Considering the different statements individually and comparing the responses for the three methods (Fig. 1), it can be seen that participants rated the submission folder method more highly. The difference between the online system and the Moodle system was small, however, participant responses indicated that the online system was less complex to use, quicker to learn and they would use it more frequently.

Fig. 1.
figure 1

SUS responses

Full size image

The participants’ selection of which method was preferred in response to the 10 statements listed in Sect. 2.2, was analysed by calculating the amount of times each method was selected overall for each item (Fig. 2). An interesting trend to note in the graph of the items is the high percentage of selection of the submission folder method for items two to six. Participants indicated that they found it easier to use and learn and indicated that it was more efficient and effective for submitting assignments, supporting the results of the SUS survey items. However, participants also indicated that they were more confident using the Moodle system than the online system. This contradicts the responses from the SUS survey, where the means were the same (\(\mu = 3.75\)).

The results indicate that participants would be more likely to use the online method in future. This differs from the results of the SUS survey to use the system more frequently. The submission system mean score (\(\mu = 4.00\)) was higher than that of the online system (\(\mu = 3.72\)).

The results of the last four items indicate that participants preferred the online submission method. The online system was the only method providing feedback to participants that the file had been successfully submitted. Most of the participants did not indicate completion of the submission folder and Moodle tasks during the study, thus not receiving emailed marked reports. Not receiving the emailed mark reports may be the reason for the low ratings for marking accuracy and feedback responses for the Moodle and submission folder methods.

Fig. 2.
figure 2

Comparison of the methods

Full size image

Participants were also encouraged to provide comments related to their experience of the system. Only nine of the participants provide any comments. Two participants commented on the difficulty of the URL for the online system. No link was provided for participants to click on and participants had to type the URL in after reading it on the instruction sheet (paper-based).

Participants (\(n=2\)) also commented that the Moodle system was “long and complicated”. Participants had to sign in to the Learn site, navigate to the module page, navigate to the link for the survey, select to upload a file, provide details for the file and then save the information, in order to upload the file. One of the participants commented that it was “easier to save in the submissions folder as you can see that it is saved”. This also alludes to the need for some form of confirmation that the file has been submitted. One participant commented on the feedback provided in the mark report. In particular that the report only indicates that a task was done incorrectly but not how to correct it.

4 Conclusions and Future Work

The aim of the study was to identify students’ attitudes towards the different methods of submitting assignment documents. Although the assignments were specifically for the CFM as part of an ICC at NMMU, the results are useful for any course where files are required for submission.

The overall results indicated a preference towards the submission folder method. This method, according to participants, is easy to use, effective and efficient. However, the online submission system provided participants with a greater level of satisfaction that the file had been submitted successfully. Overall, participants preferred the simplicity of the submission folder method, while wanting feedback that the file had been submitted (online system). A process that requires too many steps in order to submit the file (Moodle method) is not efficient or desirable for participants to use.

The immediate feedback provided by the online submission system resulted in participants being more satisfied with the marking accuracy and feedback of the online system. The only difference between the mark reports provided by the three submission methods is that, for the Moodle and submission folder methods, the marking was manually initiated, which many participants failed to do.

The study also revealed that an AGS would be beneficial if feedback could be provided to students regarding weekly assignments. Many students participating in the study enquired whether other assignments, not part of the study, could be submitted for assessment so that feedback can be received. In addition, if assignments are assessed on a weekly basis, the scores could be used for summative assessment and instructors would have feedback on whether or not students were achieving the learning outcomes or lacking in certain skills. Feedback provided to students on how to correct errors or at least more detail regarding what the problem may be, would also be beneficial.

It is acknowledged that it would be expected that students would prefer the submission folder method as they have used it more often, and that the results could be anticipated. This would be from an educator/researcher perspective. The purpose of the study was to determine student views on the preferred submission method. Additional information regarding the different submission methods was revealed from the study which will be beneficial for future work.

The study was unable to evaluate the accuracy of different marking systems and was restricted to feedback provided by participants attending a weekly lab session. Future work would further investigate which methods are more effective and efficient when working off-campus, especially when considering the use of virtual private networks to use the submission folder method. In addition, it would be beneficial to determine if students prefer downloading the mark reports themselves or if they would like to receive an email with the mark report as an attachment. The detail provided by the mark report could also be investigated further.

There is also scope in the future to evaluate improvements to the online submission and marking system, including sign in screens, user accounts keeping track of assignments and allowing students to submit assignments on a weekly basis for assessment. Results from this study have indicated that the system should be kept as simple and easy as possible, while providing useful feedback.