An Inquiry into Perception and Usage of Smartphone Permission Models

Abstract

Initially, Android and iOS took different approaches to protect users’ privacy from third-party apps by means of permissions. The old Android permission model has been repeatedly criticized for its poor usability, whereas the runtime permission model of iOS received relatively low attention in the usable security community. Since October 2015, Android also implements the runtime permission model. We compare perception and usefulness of the respective permission models by three groups: users of old Android, runtime Android and iOS permissions. To this end, we conducted a survey with over 800 respondents. The results indicate that both permission types are reportedly utilized by users for decision making regarding app usage. However, runtime permissions in Android and iOS are perceived as more useful than the old Android permissions. Users also show a more positive attitude towards the runtime permission model independently of the smartphone operating system.

1 Introduction

Smartphones store and process a large amount of sensitive personal data. Until recently, Android and iOS took different approaches to protect this information from unwanted access by third-party apps. Following the terminology by Bonne et al. [4], we call these approaches old Android permission model and runtime permission model, respectively.

Runtime permission model was introduced in iOS 6 in September 2012.¹ If an installed app needs access to sensitive data for the first time, the user is presented with a permission request for this data type and can grant or refuse access. Moreover, users can adjust (grant or revoke) permissions in the smartphone’s settings. Thus, the runtime permission model allows for fine-grained control over the data access by the apps.

The old Android permission model was used by Android prior to the introduction of Android 6.0 (Marshmallow). During the installation process of any app, Android users are shown permission requests by this app. If the users do not want an app to access one or more of the required data types, they have to cancel the installation process. However, if users install the app, they permanently grant all permissions. This permission model has been repeatedly criticized for its poor usability. User studies have shown that many users do not notice and do not understand permissions. Moreover, the users are required to take an “all-or-nothing” decision at a psychologically inconvenient time point, as they are shown the permission screen after they already decided to install an app [6, 7, 9].

Maybe in response to the above critique, the old Android permission model was changed to the runtime permission model starting with Android 6.0 in October 2015. Referring to Android’s permission model before and after version 6.0 (Marshmallow), researchers have used different terminology, as presented in Table 1. We use the terminology of Bonne et al. [4] and thus refer to the Android permission model before the version 6.0 as the old Android permission model and to the permission model for version 6.0 and later as the runtime permission model.

Table 1. Various terminology used for Android permission models

It seems that the runtime permission model is considered to be “better” by both, Google and Apple. However, it is not clear whether users also perceive the runtime permission model to be “better”, and whether these perceptions differ for different smartphone operating systems, Android and iOS. The goal of this work is to investigate these questions.

Currently, some users have smartphones with the old Android permission model, whereas others already use the new one. Thus, we have a unique opportunity to compare usage and perception of permissions of these two user groups. Additionally, if we capture the same data from iOS users, we may be able to see whether usage and perception of runtime permissions are similar across both smartphone operating system. More precisely, we consider the following research questions:

• RQ1: How are different permission models reportedly used in practice?²

• RQ2: How are different permission models perceived by the users?

To answer RQ1, we consider the reported role of the old Android permissions in the installation process and the reported behavior of users when they encounter runtime permissions: how they usually react and whether they adjust permissions in the smartphone’s settings.

To specify RQ2 more concretely, we formulate the following hypotheses:

• RQ2-H1: Runtime permissions are perceived as more useful than old Android permissions.

• RQ2-H2: Runtime permissions are perceived more positively than old Android permissions.

To answer these research questions, we conducted an online survey with 864 participants: 339 users of old Android permissions, 211 users of runtime Android permissions and 314 iOS users. We found that both permission types are reportedly utilized by users for decision making regarding app installations and usage. However, runtime permissions in Android and iOS are perceived as more useful than the old Android permissions. Users also show a more positive attitude towards the runtime permission model.

Outline. This paper is organized as follows. We discuss related work in the next section, and outline study design and participants’ demographics in Sect. 3. Study results are presented in Sect. 4 and discussed in Sect. 5. We conclude in Sect. 6.

2 Related Work

User perceptions, attitudes and behavior concerning various aspects of smartphone security and privacy has been an active research topic in the last decade. Here we focus on research regarding permissions.

iOS runtime permissions have received limited research attention so far. Tan et al. [13] investigated how developer-specified reasons for iOS permission requests influence user behavior. They found that users are significantly more likely to grant permission requests when an explanation was available, even if the content of that explanation was not relevant for the app usage.

Previous research mainly focused on Android permissions. We provide an overview of research on perception of the old Android permission model and possible design improvements. We then consider research on alternative presentation forms and extensions of the old Android permission model, and finally on the runtime permissions.

Regarding the old Android permission model, users are confronted with making privacy related decisions at installation time of an app by either granting all requested permissions or aborting the app installation. This process often induces users to grant all permissions without reading them or without understanding the consequences [6, 9]. Android users seem also unaware of the frequency of apps collecting personal data e.g. regarding tracking data points [8] and of apps continuing to access smartphone’s resources when running in the background [14]. Kelley et al. [9] report that the permission display is read in general, but rarely understood. Even text-based warnings which explain the access of an app do not show a strong effect on decisions about app installations [3].

In order to improve the comprehension of applications accessing and changing data and settings on users’ smartphones, many alternative interfaces and extensions to existing permission systems have been developed [10, 11, 15].

Some research efforts tried to increase understanding and usage of permissions for decision making by providing additional information next to the permission screen. Kelley et al. [10] designed a display with privacy information to help users to make better decisions on security and privacy in choosing applications with fewer permissions. As a result, they could support users in selecting apps with less permissions. Kraus et al. [11] supplied users with statistical information: the number of permissions compared to other apps with similar functionality. Tsai et al. [15] argue that the Android permission privacy interfaces are insufficient in helping users making informed decisions about privacy desires and needs because they disregard contextual factors. Therefore, they present TurtleGuard, a privacy feedback interface based on machine-learning techniques. Overall, Android users have consistently expressed surprise about apps’ data access and a desire to have more control over it [1, 7, 16].

The first study on the adaptation of Android users to the runtime permission model was conducted by Andriotis et al. [2]. They designed an application which was installed by 50 Android Marshmallow users. This app gathered smartphone’s current information about permission settings for each installed app. Additionally, participants were asked six multiple choice questions about their understanding and perception of the new model. This study shows that the majority of users prefer the new permission system as it enabled them to better control their data being accessed by apps. Bonne et al. [4] examine reasons why Android users install or remove apps from their smartphones in the runtime permission model. The authors collected data using questionnaires and observed real app usage behavior through the use of an Android app. They conclude that requested permissions are less important for user’s app choice and that 15% of users uninstalled apps due to permissions. We further discuss the work of Andriotis et al. and Bonne et al. in Sect. 5.

To summarize, usage and usefulness of Android permissions have been a very active research topic, whereas iOS permissions have not received much attention. Although runtime permissions have been positively received by the Android users, it is not clear whether research results concerning Android runtime permissions can be generalized to the iOS runtime permissions as well, and to possible future uses of runtime permissions in other systems. We take the first step in closing this research gap by comparing usage, usefulness and attitude to permissions by the three major smartphone user groups available today: users of old Android permissions and users of runtime permissions for both, Android and iOS.

3 Method

In the following we describe the survey design, data analysis approach, the recruiting process and the characteristics of the participants.

3.1 Survey Design

The survey focused on users’ reported handling of permissions at installation and during runtime, on perception of permissions’ usefulness and on positive or negative attitude to them. It consisted of the following question groups:

• 1. Smartphone usage: OS version, duration of usage, OS of the previous smartphone³, number of self-installed apps, frequency of app installations;

• 2. Usage of permissions: important factors in the app choice process, canceling of installations due to permissions requests, handling of runtime permissions;

• 3. Usefulness of permissions and attitude towards permissions;

• 4. Demographics: age, sex, education, occupation, affinity towards technology. The latter was measured using the psychometric scale by Zawacki-Richter et al. [17] (in German). This scale rates eight statements by using a 5-point Likert scale (from “disagree” to “agree”). The statements cover experience, competence, attitude, knowledge, interest and acceptance towards technology.

In the questions about usage of permissions users were first shown a screenshot of the respective app store and asked to indicate which interface elements are important to them when choosing an app. The participants were shown the list of interface elements (e.g., app name, app size, price, reviews) in randomized order and asked to order the elements by importance. For the users of the old Android model, this list contained the item Requested permissions and was used to establish the role of the old Android permission model in the app choice process. We also asked whether users sometimes cancel installation of apps and for what reason. Afterward, we explicitly asked about canceling of app installations due to permissions.

With respect to runtime permissions, we first showed to the users example situations that arise if an app asks for permissions. We asked whether the users are familiar with similar situations, and how they usually react to them (latter as a free-text question). Furthermore, we asked users of runtime permissions whether they have ever changed permissions in the settings of their smartphones.

The usefulness of permissions was assessed with the statement “I find permissions useful”. It was rated on a 5-point Likert scale from 1 = “disagree” to 5 = “agree”. To assess users’ attitude towards permissions, they were asked to complete the following statements on a 5-point Likert scale (from 1 = “negative” to 5 = “positive”):

• My attitude toward permissions is generally ...

• My overall experience with permission requests is ...

We conducted several pretest runs with Android and iOS users during the survey design process. We first tested individual questions with users of both operating systems and adjusted them accordingly. Finally, the complete questionnaire was tested by five Android and two iPhone users.

3.2 Data Analysis

We calculated Chi-squared tests ($\chi ^2$) for nominally scaled variables or Analyses of Variance (ANOVA) for interval scaled variables, respectively. Significant differences are indicated if $p<.05$. Because of the large sample size, even small effects reach statistical significance. Therefore, we also report $\eta ^2$ as estimate of effect size in ANOVAs, and Cramer’s V ($\varphi _c$) for $\chi ^2$ tests. According to Cohen [5], effect size is considered to be small at $\eta ^2=0.01$ or $\varphi _c=0.07$, medium at $\eta ^2=0.09$ or $\varphi _c=0.21$, and large at $\eta ^2=0.25$ or $\varphi _c=0.35$.

The reliability of the scale “affinity for technology” was assessed by Cronbach’s $\alpha$ as a measurement for internal consistency. With a Cronbrach’s $\alpha = 0.85$, the reliability is good.

The survey also contained open-ended questions, which were categorized using MAXQDA. We applied an inductive approach, meaning that categories were derived from the data material. A given answer could be assigned to more than one category. As the most free-text answers were very short and unambiguous, the categorization codebook was compiled by one researcher. The resulting categories were then discussed by the research team, and thereafter one researcher coded all answers.

Table 2. Demographics of the participants ($N=864$), $\sigma$ denotes standard deviation. Some values are missing in the dataset, therefore values do not always add up to 864.

3.3 Participants

The questionnaire was available online for 30 days in October 2016. It was approved by the data protection office of the Friedrich-Alexander University of Erlangen-Nuremberg (FAU). We advertised the study on the mailing lists of the economics and the social sciences departments and at the official Facebook group of the FAU. To avoid self-selection and priming issues, the recruitment message stated that the study was about smartphone usability.

The average completion time of the questionnaire was fifteen minutes. The users did not receive any compensation for participation. Overall 1164 people took part in the study. 208 answers were sorted out because these participants did not complete the survey. Additionally, 92 participants either did not have a smartphone with Andorid or iOS, or did not provide their Android version in the questionnaire.⁴ Both types of participants were not asked any further questions. This yielded a dataset of 864 utilizable responses.

Consistency of Smartphone Usage. As described in Sect. 3.1, we asked participants how long they have been using their current smartphone, and which kind of smartphone they had before (if any). We did this in order to identify users that recently switched from Android to iOS or vice versa, as we were afraid that they might confound both permission systems in their answers. However, this fear was not justified by the data, as we discuss below.

We define users that consistently used operating system OS $\in \{$Android, iOS$\}$ if and only if they satisfy one of the following conditions:

• currently use OS & it is their first smartphone;

• have been using OS since 2014⁵ or earlier;

• currently use OS & previous smartphone had OS.

Out of 864 users, 760 users (88%) used their operating system consistently. Comparing their answers with the answers of all 864 users, we found no statistically significant differences in any of the results that we present in Sect. 4 (mostly, the descriptive statistics were exactly the same). Thus, we conclude that the answers of “inconsistent” users did not influence the results. This may be due to the low number of these users, or possibly consistency of usage is not important for our research questions. In any case, in the following we present results based on the dataset of all 864 users.

Demographics. The sample characteristics are presented in Table 2. Two-thirds of participants are female, most of them are students in the lower semesters, before Bachelor’s degree. On average, their affinity for technology is low (under 3).

Our research questions include statistical comparison of the three user groups with the permission model as independent variable and usefulness of permissions (RQ2-H1) and attitude towards them (RQ2-H2) as dependent variable. Therefore, we need to make sure that the results of the analysis are not confounded by the differences in the demographic characteristics of the three groups. To determine whether the three groups differ in their demographic characteristics, we calculated the corresponding statistic measures with permission model as between-subjects factor and a demographic characteristic as dependent variable.

In rare cases, some values are missing in the dataset, therefore we report sample sizes for each statistic result separately. The groups were similar in their average age ($F(2,860)<1$), educational degree ($\chi ^2(6,N=862) =8.315$, $p=.216$, $\varphi _c=.07$), and occupation ($\chi ^2(4,N=861) =6.084$, $p=.193$, $\varphi _c=.06$). However, their affinity towards technology significantly correlated with operating system (and thus permission model) they used, $F(2,863)=17.150$, $p<.001$, $\eta ^2=.04$. It was lowest for participants with the old Android permission model, $p<.001$, whereas participants with runtime Android permissions or iOS did not differ, $p=.233$. As men in the sample were more affine towards technology than women, $F(1,862)=221.227$, $p<.001$, we also observed a small but not significant effect of sex, $\chi ^2(2, N=863) =5.256$, $p=.072$, $\varphi _c=.08$.

Because this difference in the affinity towards technology might confound the effects of the permission model, we controlled for this variable statistically. In the following, we calculated Analyses of Covariance (ANCOVA) with affinity for technology as covariate. Because this was not possible for Chi-squared tests, we calculated the correlation between the affinity for technology and the respective dependent variable to assess whether there was a confound. This was not the case (all $r<.10$).

App Usage. All three user groups have similar experiences with app installations: most installed 30 or less apps. The majority of participants installs apps several times per month or per year. Overall, users of old Android versions install less apps than the other groups.

4 Results

We present the findings of our study according to the two research questions in the following.

4.1 RQ1 – Usage of Permission Models

To answer RQ1, we consider the role of the old Android permissions in the installation process, and the reported behavior of the users when they encounter runtime permissions: how they usually react, and whether they adjust permissions in the smartphone’s settings.

Usage of Permissions for Installation Decisions. When asked to place some elements of user interfaces of the respective app stores in the order of their importance for app choice, the three user groups reported similar behavior. Top 3 elements in all groups were “Price”, “Reviews” and “App Description”. Both groups of Android users put “Permissions” at the fourth place (iOS users do not have this interface element in their app store). 9% of old Android permissions users and 10% of runtime Android users put “Permissions” into the first place. We also asked the participants an open question about additional factors that they consider when choosing an app. Permissions were mentioned by 5% of iOS users, including three users that never used an Android smartphone.

Waiving of App Usage Due to Permissions. When asked whether they have ever canceled app installations or usage because of permissions, 45% of old Android permissions users, 46% of new Android permissions users, and 31% of iOS users answered in the affirmative. The close similarity between the answers of the users of old Android permissions and runtime Android permissions may be due to the fact that, according to the survey results, all but 11 Android users encountered both permission models.

In an open-ended question about the canceling reasons, users across all permission models most often reported that they waive app usage when there is no understandable reason why an app should access certain resources. Users expressed concerns about data security and privacy, and criticized lack of transparency why apps need certain permissions. Users often hesitate to use apps if they require permissions that are obviously not related to the functionality, e.g., a calculator app requiring access to the contact list. Furthermore, the participants mentioned specific sensitive data types which, when being accessed by apps, lead to their non-usage or deinstallation. Permissions such as location, photos and contact lists were mentioned most often by users of all permission models.

Usage of Runtime Permissions. Survey participants were shown examples of runtime permission requests and asked whether they are familiar with such situations (see Sect. 3.1). Overwhelming majority of runtime Android and iOS users (99%) answered in the affirmative.

The participants were subsequently asked an open question about how they usually react in these situations. Their strategies can be subdivided into three categories (see Table 3). The majority of runtime permission users say that they usually take situational decisions. This means that they decide on whether to allow or decline permissions based on the necessity for the app or depending on the permissions type. Some users feel that some permissions, such as location, camera, contact list or microphone, are more sensitive than the others. Some users reported the strategy of first denying all runtime permissions and then granting them if the app does not work as expected.

A notable minority of users (around 20%) report that they usually grant permissions. In this case, some users commented that as they download apps from the official stores, they trust these apps.

Table 3. Usual behavior towards runtime permissions requests (211 runtime Android users and 314 iOS users; percentages not always add up to 100% due to rounding)

Users were furthermore asked whether they use the possibility to change their permission decisions in the settings of their smartphones. Strong majority of users, 83% for iOS and 71% for Android, answered in the affirmative. The difference between user groups may be due to the fact that the Android users have not fully adapted to the new permissions model yet.

4.2 RQ2 – Perception of Permission Models

Both hypotheses formulated in Sect. 1 could be supported:

• RQ2-H1: Runtime permissions are perceived as more useful than old Android permissions.

• RQ2-H2: Runtime permissions are perceived more positively than old Android permissions.

The usefulness of permissions was assessed by asking the participants to rate the statement “I find permissions useful” on a 5-point Likert scale from 1 = “disagree” to 5 = “agree”. The results are shown in Table 4. The permissions model correlates significantly with the perceived usefulness of the permissions, $F(2,860)=5.987$, $p=0.003$, $\eta ^2=0.014$. In particular, users with runtime Android or iOS systems rated permissions similarly useful, and as more useful than users with the old Android system.

Table 4. Results of participants’ ratings on a 5-point Likert scale from 1 = “disagree” to 5 = “agree”($N=864$)

Table 5. Attitude and experiences with permissions on a 5-point Likert scale from 1 = “negative” to 5 = “positive”($N=864$)

Attitude to permissions was assessed by two items, both rated on a 5-point Likert scale from 1 = “negative” to 5 = “positive”. The results are shown in Table 5. The permission model is significantly correlated with the attitude, i.e., with the general attitude, $F(2,860)=15.309$, $p<0.001$, $\eta ^2=0.034$, as well as with the overall experience, $F(2,860)=5.233$, $p=0.006$, $\eta ^2=0.012$. Although, users perceive permissions as slightly negative on average, this emotion is weaker for runtime permissions, nearing the neutral attitude. Moreover, the overall experience with permissions is reported as neutral, but with more positive reaction from the runtime permissions users.

5 Discussion

According to the reported usage of permissions, both permission types are utilized by the users when they decide on installation and usage of apps. Old Android permissions seem to play an important role in the app choice process for a notable amount of Android users: they put permissions at the fourth place in the app choice process (after price, reviews and app description), and almost half of them (45%) reported that they canceled app installations because of permissions.

Users of the runtime permissions report that they cancel usage of apps that request unreasonable (from the user’s point of view) permissions. The request for unreasonable permissions could diminish trust into the app. Another possible explanation might be that if the app is not important for the users, they may decide that additional effort required for management of runtime permissions is not worth the benefit they get from the app. However, users of runtime permissions cancel app usage less frequently. This may be due to the fact that many apps still work as expected if the users are free to manage the permissions.

Andriotis et al. [2] found in a within-subjects study that Android users encountering both permission models prefer the runtime permission model to the old Android model. The authors used an Android app to collect data from participants’ smartphones regarding permission settings for each installed application. Additionally, participants were asked six multiple choice questions about their understanding and perception of the new model. In comparison to Andriotis et al. we used a more extensive questionnaire including also open questions, analyzed a larger user sample and included iOS users in our data collection process. Therefore, we were able to investigate the perception of permission systems independent of the smartphone operating system. However, we did not ask our participants to download an app, and therefore we could not observe their actual behavior. Our results confirm the findings of Andriotis et al. Corroborating their evidence, we find in a between-subject study with Android and iOS users that runtime permissions are perceived more useful and more positively than the old Android permissions. A more extended study that uses both, Android and iOS apps, could build on our findings and provide further insights into permission perception and usage.

Bonne et al. [4] examine how Android users of version 6.0 or higher decide on installing or removing apps from their smartphones. They also logged grant and denial rates of permissions. The authors used data from questionnaires as well as data from observed real app usage behavior through the use of an Android app. Regarding app choice, requested permissions were found to play the least important role in the survey results. This is in contrast to our survey where permissions were ranked fourth after app price, reviews and app description. However, logged data revealed that 15% of all users uninstalled apps due to permissions. In our survey, 30% of iOS and 46% of Android runtime model users stated that they at least once canceled app usage because of permission requests. As our question was formulated without time boundaries, but Bonne et al. observed their users for a limited amount of time, we think that our survey results are reasonably close to reality, corroborating the results by Bonne et al.

To summarize, we find that the runtime permissions provide users with both, the benefits of the old Android permissions (as they can decide to cancel app usage in case of unreasonable permissions requests), and with more control over apps that they want to use despite some unwelcome permissions requests.

6 Conclusion

We conducted a survey with over 800 respondents comparing perception and reported usage of the respective permission models by three groups: users of old Android, runtime Android and iOS permissions. Both permission types are reportedly utilized in users’ decision making concerning app installation and usage. However, runtime permissions in Android and iOS are perceived as more useful and evoke a more positive emotional attitude than the old Android permissions.

Our study has several limitations. We use a convenience sample, mostly consisting of students, and two-thirds of participants are female. Furthermore, the three user groups differ in their affinity towards technology (we control for the latter in our statistical analysis). Therefore, it is not clear how our results can be generalized to other population groups. Additionally, as we used an online survey, we could not assess the actual behavior of the users, but only their reported behavior.

Future work is especially needed to understand the actual effectiveness of the runtime permission model, that is, how well it prevents users from unintended installation of privacy-invasive or malicious apps.