Improving Mobile User Experience of New Features Through Remote Tests and Evaluation

Abstract

This paper presents a process to evaluate a new feature which allowed users to do gestures in a fingerprint sensor located at the bottom of smartphones. The idea was, with these gestures in the fingerprint sensor, users could perform all functions available on a typical Android navigation bar (go back, go to home screen and open the recent apps), so that the navigation bar could be hidden from all screens, allowing the users to enjoy a larger screen in all apps. The whole evaluation process was remotely performed at the end of the development process, with 115 participants receiving a smartphone containing not only the new feature to be evaluated but also a few embedded apps to collect logs, let users raise any issues they found and answer user satisfaction surveys after some time using the feature. The findings from this evaluation process were useful to refine the feature, enhance the user experience and make all stakeholders more confident about the user’s acceptance before releasing it to market.

1 Introduction

A mobile phone is a must-have consumer product for most of the population. Over the years, mobile phones became much more than just ordinary voice communicating devices to deliver a variety of rich experiences, allowing people to be always connected to friends and family, play games, listen to music, be up-to-date with any kind of news of the whole world, and others.

Given that any people in the world could be a smartphone user nowadays, deliver a great user experience is a high priority requirement for all the top smartphone manufacturers. In fact, ease of use is something that people seeks in a smartphone. Also, even customization of mobile phones is mostly done by users because they wanted to use a mobile phone easily [1].

Knowing about the relevance of the user experience for mobile phones, this paper presents a process that evaluated the user experience of a new feature called “One Button Nav” added to Lenovo Motorola mobile phones before they hit the market. The process involved a remote evaluation through log collection and pre-installed apps to let users to describe any issues they found as well as to answer surveys applied after users effectively utilized the new feature for a while.

The collected information about users, their expectations and experiences allowed designers to continuously enhance the user experience. Moreover, it helped product managers to decide whether this new feature should have been effectively included in a mobile phone, given that it measures the user satisfaction and adoption - and, as stated by [2], the value of a new feature is realized only by the users adopting it or not.

The next sections will describe the One Button Nav feature, the process applied to evaluate the feature and the major findings.

2 One Button Nav

The One Button Nav is a new feature introduced by Lenovo Motorola in their smartphones, delivered first on Moto G family in 2017 [3]. Previous models from Lenovo Motorola have already been shipped with a fingerprint sensor, but that sensor was only utilized to authenticate the user as an alternative for passwords, PINs and drawn patterns to authenticate the user and unlock the device. Figure 1 shows pictures of Moto G4 Plus and Moto G5 Plus, highlighting the fingerprint sensor at the bottom of them.

Fig. 1.

(a) Moto G4 Plus and (b) Moto G5 Plus. Fingerprint sensor in both devices is highlighted.

The One Button Nav takes advantage of the fingerprint sensor built at the bottom of the latest Lenovo Motorola smartphones to enhance it and give the ability to make gestures on the sensor itself. Thus, the idea is to have specific gestures available on the fingerprint sensor to completely replace the typical Android navigation bar and its icons (back, home and recent apps, shown in Fig. 2), increasing the useful area of the display. Although the concept of this feature is very simple, all users need to learn new gestures to use existing functionalities in a different way [4]. And, as highlighted by Nayebi et al. [5], easy to learn is a characteristic that users appreciate in mobile apps and was, in fact, one of the major objectives of the One Button Nav development.

Fig. 2.

Typical navigation bar from Android [6].

A few screenshots of the One Button Nav tutorial are presented in Fig. 3.

Fig. 3.

One Button Nav screenshots.

Table 1 summarizes all the available One Button Nav gestures.

Table 1. One Button Nav gestures.

One of the greatest advantages of One Button Nav is that it can completely replace the Android navigation bar, so that this bar can be hidden, giving to the users some more useful space on the screen – and, according to the findings from Liu and Liang [7], screen size is a key factor that people adopt to decide to buy a new smartphone.

3 User Experience Evaluation

Considering the concerns mentioned in the previous sections about the user experience, 115 participants from all over the world were recruited to test a new mobile phone containing the One Button Nav for three months. By default, One Button Nav was disabled, but participants were invited to enable it and use it as much as possible. The test devices were configured to automatically collect logs (especially errors and bugs) as well as to trigger some user satisfaction surveys if the user activated the feature and used it for a while. Participants could also manually report issues they faced through a bug report app pre-installed in their devices. Additionally, an online forum was created to let participants freely express and share with the others their opinions about the One Button Nav.

Some weeks after each survey was applied and its results analyzed, users were invited to change their devices for a newer, improved hardware and software containing enhancements based on their feedbacks. To sum up, during those three months, 62 issues related to One Button Nav were manually raised. After the development team analyzed these issues, this number decreased to 22 valid – from all issues, 41.9% were duplicated, 8.1% unreproducible, 8.1% considered invalid, 6.5% cancelled by the development team, 6.5% decided not to fix, 22.6% unresolved and 6.5% corrected for the new versions. Figures 4 and 5 shows some charts summarizing how the issues were handled.

Fig. 4.

One Button Nav issues resolution (number of issues per resolution).

Fig. 5.

One Button Nav issues resolution (distribution).

Issues classified as “unreproducible” and “invalid” were mostly related to performance due to the mobile phones used in this study had some apps listening to events and collecting logs all the time. As end user devices don’t have these apps, these performance issues won’t be faced by them.

“Business decision not to fix” issues were related to incorrect gestures performed in the fingerprint sensor. For example, when the user tried to open all apps by swipping from left to right but the finger was a little far from the sensor or not totally touching it, so that the One Button Nav was not able to effectively identify the performed gesture.

Cancelled issues were related to some misunderstanding of how the fingerprint sensor works or the usage behavior. For example, when setting up the authentication methods of the device, users can select a combination of fingerprint and password to unlock the device after restarting it. However, some users forgot about this behavior and raised an issue because the device did not unlock only with fingerprint when restarted.

Unresolved issues do not mean exactly that nothing was done. During this study, 14 issues were marked as unresolved, although 10 of 14 (72.4%) were related to expected behavior of initial design of the feature. For example, a user had the impression that the gesture was being interpreted incorrectly, but after development team analysis it was possible to conclude that the user could be right just as gesture was incorrect and sensor couldn’t interpret. The other 4 unresolved issues were raised after the release of the final user build. As it wouldn’t be possible to fix them before the release of the device, they were moved to be analyzed in the next build updates and eventually users would receive fixes through an upcoming Play Store update.

The number of fixed issues was only four to this product. But it doesn’t mean that only four issues found were effectively fixed for it, as some of the duplicated issues were also found in similar products before, so that these problems were fixed for the other products first and then the same solution was applied to this product. Therefore, the number of fixed issues presented in the previous chart was actually the number of issues firstly found on this product (not in others) and fixed for it during the development cycle.

Some other interesting data about the usage of the fingerprint sensor and One Button Nav gestures were also collected during this study. Based on them, it was found out that, although all participants were invited to use the fingerprint sensor to unlock the device (which was the simplest gesture and not really related to One Button Nav gestures), less than 24% were effectively using it at the end of the study, as seen in Fig. 6. Additionally, Fig. 7 shows the One Button Nav gesture usage.

Fig. 6.

Adoption of fingerprint sensor to unlock the device at the end of the study.

Fig. 7.

One Button Nav gestures usage.

From Fig. 7, it is noticeable that the most common gesture performed in One Button Nav was the “Go back” to the previous screen (65.0%), followed by the “Go to home screen” (23.7%). On the other hand, the least used gesture was “Launch Google assistant (0.3%). Other interesting data about the usage of One Button Nav gestures can be viewed in Table 2. Based on these data, it is interesting to notice that, on average, users performed the “Go back” gesture about 205 times daily. In fact, the top 3 gestures most used were the ones that really replaced the ordinary Android navigation bar (go back, go to home screen and open recent apps). The small frequency of usage of turn the screen off and launch the Google assistant gestures may not be really related to users not using them, but, as there are other methods to turn the screen off (through the power button, for instance) and launch the Google assistant (saying “Ok, Google”, for instance), users may prefer them instead of One Button Nav. A deeper investigation to understand the actual usage of Google assistant and how users turn the screen off is required.

Table 2. Daily usage of One Button Nav gestures.

In addition to the raised issues and the data automatically collected about the users interacting with One Button Nav, four user satisfaction surveys were applied during the research period, as mentioned before. The surveys were composed by Likert Scale [8] questions to rate agreement/disagreement with some statements about the One Button Nav experience. The first survey collected 53 responses from distinct participants whereas in the last one 115 participants answered. The surveys were applied when the users changed their devices to newer ones containing hardware changes and when a new software version containing One Button Nav improvements was available to the users. Figure 8 shows the number of participants in each survey.

Fig. 8.

Number of participants per survey.

The survey contained questions to assess the overall satisfaction of the users with the fingerprint sensor and with the One Button Nav, as well as let users identify whether they experienced issues when using One Buton Nav gestures. Figure 9 shows the responses, among all the surveys, for the question about the overall satisfaction with the One Button Nav feature. Clearly, the positive responses were increasing over the time – in the first survey, 78% of the responses were positive whereas, in the last one, it became 83%. Additionally, the unfavorable responses decreased from 10% to 5% between the first and last surveys.

Fig. 9.

Overall satisfaction with the One Button Nav among all surveys.

Table 3 summarizes the responses for each questions, comparing the results from the first and the last survey. Among all One Button Nav gestures, it is possible to verify an increase of the user satisfaction for “Go home”, “Open recent apps” and “Launch Google assistant”. For the “Go back” gesture, there was a slightly decrease of the favorable responses, but the neutral responses were raised and the unfavorable responses were hugely decreased, possibly because users started to learn how to do this gesture.

Table 3. One Button Nav survey responses for Survey 01 (first) to Survey 04 (last).

The only gesture that the negative responses increased and the positive responses decreased from the first survey to the last survey was the “turn the screen off”. In fact, some users complained that it was because they unintentionally performed the gesture: they were holding the fingerprint button without noticing that and then got the device locked by mistake.

Figures 10 and 11 show bar charts containing the responses in the first and the last surveys, respectively, for the specific questions related to the One Button Nav gestures.

Fig. 10.

User satisfaction responses for One Button Nav gestures (first survey).

Fig. 11.

User satisfaction responses for One Button Nav gestures (last survey).

Checking all the One Button Nav gestures, all of them got value considered satisfactory.

4 Consumer Analysis

The One Button Nav was first available on the Lenovo Motorola Moto G5 family of devices, launched worldwide on August 2017. After some months of usage, a few data about how consumers (end users) have been utilizing the device became available. As seen in the chart of Fig. 12, “Go back” (66.7%) is the most frequent gesture of One Button Nav used by consumers, followed by “Go home” (17.8%) and “Open recent apps” (11.0%). For this chart, 1329003 end users were considered.

Fig. 12.

One Button Nav gestures usage (consumers).

Comparing with the pre-release study performed, data about the usage of One Button Nav gestures were similar, as shown in the chart of Fig. 13. However, the percentage of end users that are really using One Button Nav was not the same as in the study: during the user experience evaluation, One Button Nav was adopted by 23% of participants whereas, for consumers, this number was only 9.2%. A few hypotheses could explain this difference: participants of the study were more tech-savvy than end users and were invited several times to try the feature, while end users didn’t receive any advertisement about One Button Nav during the initial set up of the device. A further investigation is still required, though.

Fig. 13.

Usage of One Button Nav gestures (consumers vs. participants).

A deeper analysis of the consumer needs, expectations and satisfaction could also help to understand the lower adoption of One Button Nav. Yet, informal feedbacks from consumers that have been using One Button Nav gestures highlight the major benefit of the feature: more screen space.

5 Concluding Remarks

This paper presented a remote user experience evaluation performed to assess a new feature added to recent smartphones that let users make gestures in a fingerprint sensor to replace the default Android navigation bar, giving to the users more screen space. The conducted evaluation was helpful to identify issues and possible enhancements during the development stage, as well as brought some insights about the most common interactions (go back, go to home screen and open recent apps).

As future work, authors would like to apply a survey to consumers to measure the user satisfaction with One Button Nav and compare the numbers with the ones from the remote user experience evaluation.