1 Introduction

Mobile applications (apps) are becoming increasingly important tools in the delivery of education and training in colleges and universities. In addition to voice communication, mobile phones also allow the transfer of data that is useful for delivery of educational content (Valk et al. 2010) and broaden the availability of quality educational materials through decreased cost and increased flexibility . Mobile learning is the point at which mobile computing and electronic learning intersect to provide anytime, anywhere learning experiences for the students (Majumder and Basu 2010). Mobile learning is widely being adopted by colleges and universities to support digital learning initiatives (Seppälä and Alamäki 2003). At Fiji National University (FNU), as students continue to adopt smartphones and tablet devices at an increasing rate, the university is exploring opportunities to develop applications to serve the needs of these students.

FNU is one of the largest universities in the South Pacific region with campuses and centres located all over Fiji. There are a few courses in management, education and information systems offered through distance and flexible learning (DFL) mode to those students who are either working full time or cannot attend classes due to distance constraints. Many of the students studying through DFL modes are staying in remote locations and consequently can’t travel to campus and access the required facilities for learning. Given that the cost of bandwidth is high and there is a lack of infrastructure to facilitate distance and flexible learning, mobile learning sounds potentially a very viable and promising solution for the accessibility constraints experienced by remote students. Fiji has a mature mobile market with a high penetration rate of mobile phones; hence, most students already own mobile devices. At the beginning of 2015, the MLearn pilot project was trialled to support teaching and learning in DFL mode. The project team developed a mobile app known as MLearn that provides students with access to lecture notes, tutorials and other important information about their courses.

While mobile devices are becoming more sophisticated, applications developed to support teaching and learning in colleges, and universities are facing several challenges (Kim et al. 2006). Usability is one of these challenges. After launching the MLearn app, the development team started receiving number of complaints from the students such as that the app was difficult to use or that it was difficult to find certain features in the app. In response to the dissatisfaction expressed by the students and their reluctance to adopt the MLearn application, an in-depth usability evaluation of the mobile learning application at FNU was carried out to provide recommendations for improving the application. The term usability is not new in today’s system development. The traditional view of usability that is popular among software developers is the attributes of the software that make a product easy to be used. Usability has been recognised in the literature as an important factor that significantly affects the success or failure of any application (Bevan 2009). Usability testing of software applications developed for mobile devices is an emerging research area that faces a variety of challenges due to the unique features of mobile devices such as small screen size, limited input capability and changing user context (Harrison et al. 2013). Usability studies have been recognised as a fundamental tool in evaluating mobile apps (Zhang and Adipat 2005). However, very few usability tests have been developed for mobile learning applications.

This study aims to elicit user perceptions and overall attitude towards the mobile learning application at FNU. A review of the literature indicated that three elements are relevant to assessing the MLearn application, ease of use, usefulness and satisfaction (Ali 2013). A usability evaluation framework was established for assessing the MLearn application. The test was conducted using 30 students studying in DFL mode. The findings presented are based on an analysis of their response to questionnaires and in group interviews. This chapter commences with a systematic literature review on the usability of mobile apps. This is followed by the details of results from the usability experiment and recommendations for improving the system. Finally, the chapter concludes with future research work to be done in this area.

2 Literature Review

2.1 Mobile Learning

In the past few decades, mobile devices and related technologies have grown tremendously and are now prominently being used in many fields such as entertainment (Leong et al. 2011), health (Ducut and Fontelo 2008), and finance (Donner and Tellez 2008). Mobile phones have also found their way into learning and the education sector. Mobile learning is a rapidly growing technology which is available anytime, anywhere according to the convenience of learners (Traxler 2009). Educational institutions in the Asia-Pacific region and around the world have started to use mobile technology to facilitate learning in new and innovative ways. Some of the examples from the pacific region include mobile learning at the University of the South Pacific, mobile learning organiser at Massey University and SimPharm at the University of Otago. Mobile learning is specifically relevant to students at Fiji National University who need to study primarily from a distance, as it provides flexible access to learning opportunities for those who are distant and bound to time constraints (Rosli et al. 2010). For the purposes of this project, we define mobile learning as any sort of learning supported by mobile and other handheld devices that is not dependent on location. The learner benefits from the opportunities offered by mobile technologies such as learning anytime as there are no constraints imposed by the location of the student.

While initiative is being taken to implement mobile learning across FNU, it is necessary to understand the potential benefits and drawbacks of using these technologies for enabling access to learning to ensure that mobile learning initiatives are being implemented effectively. Asabere (2013) and Chandhok and Babbar (2011) discussed some of the benefits of using mobile learning technologies which include reducing the barriers imposed by geographical constraints, enabling learning at any place and at any time and helping students develop a self-centred learning pedagogy . Asabere (2013) and Sharples et al. (2009) highlighted the potential challenges for learners and instructors of mobile learning including;

  • Mobile learning a gap between technically sound students and non-technically efficient students.

  • It is highly dependent on a platform of network resources.

  • It can create a sense of isolation among students and instructors.

Notably, much research is occurring in the field of mobile learning around the development of innovative applications that can enhance teaching and learning. For example, Shuib et al. (2015) designed a simple Intelligent Mobile Learning Tool for Grammar Learning (i-MoL) which could help to facilitate English learning among students in non-English speaking countries; Boyinbode and Ng’ambi (2013) built a mobile lecturing tool named MOBILect that enabled students to efficiently use educational resources as lecture podcasts, while Bartel and Hagel (2014) presented a game-based mobile learning tool called eMgage. In contrast, Wishart and Green (2010) discussed lecturers’ use of Edutxt to provide feedback to students. The future prospects of mobile learning are astonishing and can be taken up by universities and higher education institutes to enable students to foster knowledge at their convenience . As a result, distance may become less of a hindrance for the provision of education. Providing personalised knowledge, network access, and infrastructure, as well as dealing with technologically naïve students, however, remains a huge challenge (Yousef et al. 2015).

2.2 Usability

The term usability was first used in the 1980s and is used to evaluate the performance and acceptance of products and systems (Wei et al. 2015). ISO 92411 (1998), The International Standard Organisation (ISO), defined usability as “Ongoing product generally specified by users which aim to achieve a set of precise goals in a context of use pertaining to effectiveness, efficiency and satisfaction”. Nayebi et al. (2012) defined usability as “the capability of a software system to be understood, learned, used and liked by the user under specified conditions”. Human computer interaction (HCI) researchers have recognised that to produce computer systems with good usability, it is important to understand physiological factors on how humans operate (Harrison et al. 2013). Usability attributes help assess the user-friendliness and quality of certain products. Nielsen (1994) identified five generic attributes of usability: efficiency, satisfaction, learnability, memorability and errors, which are widely used in usability evaluation. Efficiency refers to resources expanded in relation to accuracy and completeness with the user to achieve goals. Satisfaction refers to freedom from discomfort and positive attitudes towards the use of the product. Learnability states that a system should be easy to learn so that the user can rapidly start getting work done using the system. Memorability is that the system should be easy to remember so that the user is able to return to the systems after some time. Errors indicate that the system should have low error rate or that the users make few errors, while using the system, and if they make errors, they can easily recover from them.

Mobile usability is regarded as an emerging specialisation within the field of usability (Wei et al. 2015). The study of HCI for mobile devices started more than a decade ago, but there is still an opportunity for technology-driven research due to recent developments in mobile devices such as iPhones and smartphone . Achieving high-level user satisfaction is critical to the success of mobile apps; hence usability testing of mobile apps is a mandatory process to ensure that mobile apps are practical, effective and easy to use (Ali 2013; Kumar and Hussein 2014;). The advent of mobile devices has presented new usability challenges that are difficult to model using traditional methods of usability. Some of these challenges include:

  • Mobile context: the user is not tied to single location thus simulating the actual environment for testing is a daunting task.

  • Small screen: mobile devices have very limited screen sizes.

  • Data entry methods: the input methods available for mobile devices are different from those for desktop computers.

Mobile usability evaluation consists of methodologies for measuring the usability aspects of a system and identifying specific problems. The commonly used usability evaluation techniques are field evaluation and laboratory evaluation (Kjeldskov et al. 2005). In field evaluation, the product is evaluated in the actual context in which it is used. Using this approach, the dynamic mobile context is taken into consideration, which will be difficult to simulate in laboratory experiments. However, the major challenge for this technique is the lack of control over the participants in the study. For laboratory evaluation, usability testing is completed in the usability laboratory, which is an environment where users are studied interacting with a system in order to evaluate the system’s usability. The tester has full control over the experiment, and they can define particular tasks to match the goal of the experiment. It becomes easy to measure the usability attributes and interpret results by controlling other variables in a laboratory environment. Usability is one of the key challenges faced by mobile learning application developers. Several usability studies have been conducted in the past for application redesign and optimisation purposes.

Gebb and Young (2014) compared mobile apps such as DynaMed, Evernote, Epocrates Rx and Mobile Prescribing Reference (MPR). These mobile apps were suggested to students at the Nursing University of Frontier to enhance teaching content for online courses and provide mobile ready clinical reference resources on handheld devices. The survey in this research included a set of questions about students’ use of mobile devices and resources. Free response questions asked students to list five apps that assisted students with learning. A list of over 20 apps were examined, and the percentage of total responses shows that almost 50% of students mentioned that having mobile devices provided them an opportunity to study at their convenience , barring time and distance boundaries. The study finally highlighted issues with the interface design of these applications and proposed methods for optimisations.

Nichols et al. (2014) ran a usability test on the Primo discovery tool which is a midsized library research tool to discover patterns in the searching behaviour of users. The researchers tested key aspects of Primo’s design and functionality based on specific research questions. A diagnostic usability evaluation was conducted to understand Primo’s usability for users. The usability study included nine test subjects within the university community and was administered by a team which included a facilitator, note taker and principal investigator. The outcome of this test helped the researchers analyse screen activity, visual expressions and verbal comments that helped to judge the performance of the tool. The study revealed several problems that users experienced (participants were confused with the Primo search functions and search limiter labels) as well as technical challenges (inconsistent results due to indexing in search tool, collection-level records from in-house digital library were not displayed correctly) which highlight the importance of usability for mobile discovery tools and also how usability can affect user perception of mobile search tools.

Kukulska-Hulme (2007) conducted a usability study on the context of education at the Open University UK in the Institute of Educational Technology’s Masters programme in Online and Distance Education (MAODE). The study introduced students to recent progress in mobile usability to make mobile content adaptable to users as well as providing the ability to report usability issues in the field of mobile learning. About 57 alumni students of MAODE were asked to complete an online questionnaire using mobile devices such as phone, smartphone, PDA or iPod. The survey shows that the use of PDAs generated large numbers of both positive and negative comments, which helped the author to address certain usability issues like physical attributes of mobile devices, content and software application, network speed and physical environment. Lu et al. (2011) developed a context-aware educational game-based mobile app, CAMEG. The authors incorporated Management Information System (MIS) course content concepts into a game and set up a virtual science park in three laboratories (Lu et al. 2012). A new version of CAMPRG was developed in 2011 to make learning more attractive to the learners, and the usability was evaluated by observing learner perceptions towards the two games. During the evaluation, about 23 teams with 3–4 students were asked to complete a usability questionnaire for CAMPRG . The questionnaire used in this research contained 11 five-point Likert-scale items (5 for “strongly agree” to 1 for “strongly disagree”). This resulted in the development of methods to improve the usability of the application.

This literature review provided evidence that usability is a well-researched area in the field of mobile apps. Looking at several studies, it is evident that there is no systematic approach to evaluating the usability of mobile apps, or more specifically, mobile learning applications.

3 Experimental Setup

This section describes the methods and process employed to carry out the usability study. The study was conducted in two phases which included a laboratory experiment and a qualitative group interview with research participants. Three categories of usability criteria were identified, ease of use, usefulness and satisfaction. Ease of use measured how easily the users can use the application, satisfaction measured the extent to which users believed that the application meets their requirements and usefulness measured how worthwhile the users felt the application was.

The users were categorised into two groups, experienced and inexperienced users. For the lab experiment, tasks were prescribed to the participants which were carried out using virtual mobile devices that included a logging mechanism. The evaluation lasted for 2 h and the average duration of participation was 30 min. The participants were advised that their task was to evaluate a mobile learning application, rather than testing their skills. The pretest questionnaire was designed to examine the experience of participants. The participants completed the prescribed tasks on their own, and no assistance was provided unless an error occurred. At the end of the session, the participants were asked if they encountered any problem during the test. For the group interview, participants were encouraged to express additional comments about the mobile apps that were not addressed in the questionnaire.

3.1 Participants

A large number of participants are required to test the usability of mobile apps. Thus, the researchers recruited 30 students studying at FNU in the distance and flexible learning mode. The participants were recruited by advertising on the university notice board, and it was voluntary for students to take part. This sample included 15 experienced users (students who have already used the application) and 15 inexperienced users (students who tried out the application for the very first time), and their participation was on a voluntary basis. The students’ age varied and it was distributed over different age groups. Table 29.1 shows the participant age distribution.

Table 29.1 Participant age distribution
Full size table

3.2 Task and Procedure

The tasks included each user going through the application and performing a number of subtasks. The completion time for each task was recorded, and the entire operation was captured using the logging mechanism in the computer. The tasks are detailed in Table 29.2. MLearn sample interfaces are presented in Fig. 29.1.

Table 29.2 Task list
Full size table
Fig. 29.1
figure 1

MLearn sample user interfaces

Full size image

3.3 Documentation and Design

The documentation for the usability testing was prepared, and instructions were given to participants to read prior to testing. The instructions outlined the aim of the research and the steps the participants needed to complete as part of the testing session. There were 12 close-ended questions in total that the users had to complete for the evaluation. Four questions were derived from each of the given usability criteria, and each of the questions was rated on the scale 1–5, (1) very poor, (2) poor, (3) neutral, (4) good and (5) very good. Table 29.3 provides details of each of the usability criteria with key concepts and explanations. Questionnaires were derived using the key concepts .

Table 29.3 Usability criteria
Full size table

3.4 Testing

The testing was conducted in two phases. For the first phase, the testing was carried out in the lab with students going over the tasks independently. Since there were a number of tasks to be performed, the users were prescribed with different task orders so that the learning curve is evenly distributed. The students went over the tasks a number of times before completing the questionnaire. For the second phase, group interviews were conducted. Discussions were held separately with each of the groups to ascertain their views for each of the criteria and key concepts. Students were given an opportunity to provide feedback on what they felt should be improved in the application.

3.5 Group Interview

This is an informal method for collecting in-depth information regarding the needs, judgements and feelings of typical users about an application (Dumas and Redish 1999; Nielsen 1994; Rubin 1994). In group interview users discuss selected topics, such as the different functions and features, with the assistance of a moderator, and then identify issues during their interaction in their respective groups. After the participants completed the laboratory test, they were invited to the group interview. In this group interview, the research team had discussions with experienced and inexperienced users separately. Each of the usability criteria was discussed and the group’s opinions were noted.

3.6 Data Reliability

Cronbach’s alpha is a measure of the intercorrelation of items (Cronbach 1951). Cronbach’s alpha is the most common form of internal consistency reliability coefficient (DeVellis 2012). If alpha is greater than or equal to 0.6, then the items are considered unidimensional and may be combined in an index or scale. Alpha equals zero when the true score is not measured at all, and there is only an error component. A Cronbach’s alpha test was conducted using the data obtained from the 30 students to assess the dimensionality of the measurement scale. Scale reliability was assessed in term of items to total correlation, and Cronbach’s alpha was used to determine the internal consistency of the measurement scale. The results confirm that all measurements exhibited high reliability with coefficient alpha ranging from 0.83 to 0.95 exceeding or approaching the acceptable level of 0.70.

4 Analysis of Results

The study sample consisted of 30 students who successfully completed the survey . For the closed questions, the mean and the standard deviation were calculated for each of the groups, and then a chi-square test (X 2) was carried out to identify potential differences between the user groups. The analysis focused on the following aspects of the usability criteria: ease of use, user satisfaction and usefulness. The data gathered is summarised in Table 29.4.

Table 29.4 Statistical analysis of data
Full size table

Ease of Use

Measures how simple or easy it is to use the system. Table 29.4 shows the statistical analysis of results for each of the groups: overall we can see that Group 2 with a mean of 2.47 has a lower score than Group 1 which means that the experienced users find the system easy to use, compared to new users. In addition, the standard deviation for Group 1 is 0.19 which illustrates that evaluation in Group 1 was more consistent and less variable than Group 2. These were expected as experienced users have used the system before and are familiar with the application. In addition, we analysed the closed-ended questions. Users from both the groups agreed that the system was user-friendly and very similar to other mobile apps. The difference is not statistically significant (X 2(2) = 3.81, p = 0.14). Both groups of users are satisfied that meaningful icons have been used by the application with 67% for inexperienced and 73% of experienced users selecting good and very good. For the question on navigation, 80% users from both groups rated it from poor to neutral. In group interviews, for the question regarding navigation, the users pointed out that the app should have a button to move the screen back and forth rather than relying on the device feature to move the application back. New users pointed out that it is difficult to attempt the tutorial for the first time, and they suggested that some help should be provided. Users in both groups generally raised concerns about navigation and offered recommendations for improving this.

Satisfaction

Measures the extent to which the users believe the application meets their requirements. As illustrated in Table 29.4, the descriptive statistics indicate that Group 2 has the lower mean score of 2.23, which suggests that inexperienced students are less satisfied with using the application. To the question regarding application appearance, both groups of users were satisfied overall with screen colour and font. Most (85 %) of the experienced and 78% of the inexperienced users selected good to very good, and there was no statistically significant difference between them (X 2(2) = 2.43, p = 0.11). A third (33 %) of experienced users selected between good and very good for whether terminologies, fields and labels are used consistently or not, while 54% of inexperienced users selected neutral. While the inexperienced users provided differing views on the question about information provided on the different screens, 66% of the experienced users rated this between poor and neutral. In group interviews, the users from both groups raised concerns that the system should show the status of the download while downloading files using this application. From this analysis, we can assume that experienced users have higher expectations from the app as compared to inexperienced users. There were few concerns raised by experienced users, and they mostly requested new features such as messaging between lecturers and students, which highlights the need for making the application more interactive .

Usefulness

Measures how useful the users feel the application is. As illustrated in Table 29.4, the descriptive statistics for Group 1 and Group 2 indicate that both the groups have a close mean score ranging from 2.9 to 3.1. This suggests that both groups have rated the app between neutral and good. The users in both groups were happy with the features provided by the system and believed that most of what is needed is available to them. Approximately 40% of users in both groups selected good or very good and no low statistical differences between them (X 2(2) = 2.41, p = 0.09). Both groups also suggested that the system is efficient, and the tasks can be easily established. The inexperienced users have not used some of the functions, and there was a consensus among both groups that there is a need for help functions that would have basic documentation on how to use the system. In the group discussion , the users pointed out that while they attempted using the tutorial, they should be able to review what they input before the final submission. Users also suggested that there is a need to incorporate help and documentation that provides brief instructions on certain important features (Fig. 29.2).

Fig. 29.2
figure 2

Graphical representation of mean score

Full size image

5 Discussion

Mobile usability is a new area of research, and there is very little published literature on the usability studies of mobile learning applications; the relevant journals and conferences are not more than a decade old. Mobile learning has overlapping concepts with eLearning and mobile human-computer interaction (HCI); thus borrowing techniques and frameworks from these fields have been the starting point for usability studies of many mobile learning applications. As development in mobile learning progresses, these borrowed tools and frameworks are no longer adequate. There were several challenges faced during this research, including those inherent to usability studies of mobile learning applications such as small screen sizes, intermittent connectivity and other associated human factors that could not be addressed using the previously mentioned techniques. As users move across different devices with different configurations, several challenges arise due to the changing environments and device capabilities. Current usability techniques are suited for static and well-defined contexts. Mobile learning contexts are, however, often impromptu and difficult to observe, predict and simulate. Users often use different devices and different patterns of assessing the application, thus making it difficult to carry out usability studies. The physical constraints of mobile devices, especially small screen size and resolution, significantly affect the usability of mobile apps. Reading from small screens is aesthetically unpleasant. Small buttons and touchscreens limit the input capability and increase human errors. The result of usability assessment is largely dependent on the use of different input and output methods. Reliability of the data collected was another issue that the evaluation team encountered. The team decided to apply the Cronbach’s alpha, which is one of the most popular reliability statistics in use, and we, therefore, believe it is well suited to the needs of this research.

The results obtained are very useful in improving the system; all the users applied themselves diligently to the prescribed tasks and provided excellent feedback documenting their experiences and opinion on many aspects of usability as they interacted with the MLearn application. The analysis was conducted on closed-ended questions and group interviews. The research team measured the divergence between experienced and inexperienced users. Mean and standard deviation were computed as an indicative measure of user’s consensus towards the usability of the application. Chi-square was used to determine the association between the two sets of data. The usability test results show that usability problems are few and minor, and there were also differences in consensus between experienced and inexperienced users. Based on the usability problems highlighted, the research team provided the following recommendations to MLearn development team to improve the application in terms of ease of use, satisfaction and usefulness.

Navigation

The MLearn application should include buttons to move from one interface to another. Currently, the system relies on the feature provided by mobile devices which is largely dependent on different brands of mobile devices

Help feature

Although mobile apps are rarely seen with help features built in, the research team believes that it is important for new users to have this option available while using this application

Chat feature

The application should provide a chat feature for students to interact with their lecturers and also for group discussion as this would make learning more interactive

Student involvement

MLearn development team should seek avenues to have the student voice heard by the MLearn team to influence the future development of this application

Collaboration

Researchers should collaborate to develop additional usability testing protocols for mobile learning and publish results in journals and avenues that can be easily accessible for researchers working in this area

6 Conclusion

Growing popularity of mobile devices has led to a number of mobile learning applications being developed and used. However, the inherent nature of mobile devices such as small screen size and limited input capability require special attention to the usability aspect of these applications. The case study presented here demonstrates that usability is an important tool for improving the user perception of a mobile learning application. The usability criteria used in this study included ease of use, satisfaction and usefulness. As part of the study, an evaluation of usability was conducted using 30 students studying through a distance or flexible learning mode, which included experienced and inexperienced users. Comprehensive analysis was conducted on the results obtained. Based on the results of the evaluation, ways to improve the application usability were proposed. Future research is needed to further refine the usability evaluation methodologies for mobile learning applications; however, this study serves as an incremental step in the right direction.