Keywords

1 Introduction

Organisations are increasingly at risk to cyberattacks designed to manipulate users’ behaviour to create and exploit cybersecurity vulnerabilities [4, 17, 101]. This often involves attackers imitating legitimate channels of communications to prompt and trigger insecure behaviour amongst users that can result in their and or an entire organisation’s security being compromised [6, 28, 44, 45, 68, 81]. To address the increasing risk posed by cyberthreats, many organisations invest in and apply technical solutions such as firewalls, anti-virus software, and other tools for monitoring IT systems to maintain security [34, 43, 53, 70, 74, 86, 98]. Despite these efforts, recent studies have emphasised that technical approaches alone are not sufficient [49, 50, 54, 78, 96] and organisations continue to be susceptible to cyberattacks [52, 71]. This has led to increasing calls for organisations to address individual and organisational factors to maintain their security [74, 102]. To achieve this, organisations design security policies to manage users’ behaviour and encourage safe and secure usage of their IT systems [16, 34, 51, 59, 74, 86, 87, 98]. However, this approach is also insufficient as users’ do not always comply with security policies [47, 52, 62, 74, 85, 87,88,89, 96]. Furthermore, studies investigating causes of insecure behaviour indicate that these are not always related to users’ non-compliance with security policies but often overlap with other individual personal and organisational factors [8, 13, 29, 36]. Consequently, users’ behaviour continues to be frequently reported as a significant cause of security breaches [26] and there is an increasing need for organisations to discover effective ways to encourage safe and secure behaviour amongst users [12, 13, 63].

The user’s environment, including technological environment and social environment, is composed by factors that influence their behaviour. This suggests the need to bring insights from psychology, including social psychology, to the problem in order to design behaviour change interventions that will address user security behaviours [32, 33]. At the same time, technology offers a potential mode for delivering behavioural interventions. If an intervention can be automated, this can allow it to scale-up to larger user bases where other types of intervention may be infeasible. Further, it may be that the most appropriate way to intervene is via technology, for example at the moment that the user’s vulnerability is being exploited, or by changing the technological environment. Existing security technologies and management strategies already take advantage of these two benefits of technology, but, perhaps, not always in an optimal fashion.

The field of persuasive technology (or the roughly synonymous term digital behaviour intervention) is concerned with the study and introduction of technologies that change behaviour, specifically without coercion, with applications across a number of areas. A number of authors from the security domain have suggested that persuasive technologies, and persuasive techniques, may have an important role to play in security [5, 8, 22, 42, 100]. While the underlying technology substrate may sometimes be commonplace (for example email in this study), persuasive technology researchers draw upon insights from psychology in designing interventions on top of that substrate, and use rigorous scientific methods for analysis and evaluation [25, 64, 91].

One proposed behaviour change approach, much studied in recent years, includes applying behavioural nudges, in which re-design of an individual’s decision environment (‘choice architecture’) guides them to make certain choices rather than others. Often the nudge is in a form such that the individual is not explicitly aware of it. Examples of this approach are the MINDSPACE framework [35], and the SCENE framework [30] tailored for cybersecurity. Applications of nudging to cybersecurity include encouraging safer mobile device usage [9, 23, 92, 94], improving password management [55, 77], quantitative access control [69], increasing awareness and improving decision making related to social media disclosures and general privacy concerns [2, 99].

An alternative approach involves applying explicit persuasive messages. This approach is commonly applied within the persuasive technology domain, and has been demonstrated to be effective for changing individuals behaviour across a range of domains by encouraging healthy eating, increasing physical activity, participating in health and wellbeing activities and sustainable ecological behaviour [48, 73]. However, study of the use of explicit persuasive messages, such as those based on Cialdini’s well-known principles of persuasion [24], together with measures of actual effectiveness, rather than perceived effectiveness, and particularly in-the-wild, has been limited within the context of encouraging users to engage with information security. An exception is a major study of the effectiveness of the ‘social proof’ persuasive strategy [33].

In this paper, we present results from a large-scale, quantitative, empirical field study of persuasive messaging for encouraging staff in an organisation (a university) to participate in information security awareness (ISA) training. This study was conducted by incorporating explicit persuasive messages based on random assignment of Cialdini’s [24] principles of persuasion within an existing corporate communications infrastructure. Evaluation studies of persuasive messages, widely reported in the persuasive technology literature, often involve the use of perceived effectiveness as an outcome measure, based on participant self reporting measures [73]. For this study, we used actual effectiveness as an outcome measure based on the direct observable behaviour of participants in response to the persuasive messages applied during the study. This provided us with a more objective measure of the persuasive messages’ effectiveness in a real non controlled environment. Our results indicate that there are significant differences in the effectiveness of the persuasive messages used in the study and the role of individual and organisational factors. We also discovered that persuasive messages that included reference to the collaborative role of staff to safeguard the university from potential cyberthreats (aligned with the ‘unity’ persuasive strategy [24]) were more effective compared to those which emphasised the authority imposed, mandatory requirement for all members of staff to complete their training (aligned with the ‘authority’ persuasive strategy [24]).

In Sect. 2 of this paper we provide an overview of behaviour change and persuasive techniques, followed by a brief review of behaviour change interventions within a cybersecurity context. In Sect. 3 we describe our methodology, study procedure and present our research question and hypothesis. We present the results of our study in Sect. 4. The limitations of the study are discussed in Sect. 5 and in Sect. 6 we review and discuss key findings and outline our plans for future work.

2 Related Work

Interventions capable of changing individual behaviour are increasingly in demand, because of the impact of the negative consequences that may arise from an individual’s actions and decisions. For example, poor diet, lack of exercise and smoking, may result in severe health problems. Similarly, insecure usage of IT systems such as clicking on a link within a phishing email and sharing passwords may compromise security. Behaviour change interventions aim to motivate and encourage individuals towards improving their behaviour, in addition to deterring behaviours that can lead to negative and undesirable consequences [73].

2.1 Behaviour Change and Persuasive Techniques

In broad terms, human behaviour may occur as a result of either automatic, indirect (also referred to as System 1) processing and/or reflective, direct (System 2) processing of cues within the context of a given scenario or environment [19, 39, 76, 90]. Many behaviour change interventions and persuasive technology design frameworks incorporate a model of behaviour that may be used to elicit behavioural determinants or factors that may influence and change an individual’s behaviour for a given scenario [41, 65, 72]. Upon establishing how certain behaviours occur and why, it is possible to begin considering what specific techniques may be applied to bring about a desired outcome. However, it is often difficult for intervention designers to establish a suitable theoretical foundation, that provides a testable hypothesis for how and why a particular behavioural change or persuasive technique may influence and determine an individual’s behaviour [7, 66]. This is often due to the diversity and interrelated aspect of behavioural determinants that may lead to an intervention’s means of achieving the intended outcome [31].

Within the Persuasive Technology domain, a common approach towards designing behaviour change interventions involves applying persuasive messages based on principles of persuasion as defined by Cialdini [24]. Such persuasive messages may be designed to bring about changes in behaviour using either ‘System 1’ or ‘System 2’ processing, but in the case of the latter, these are intended to trigger a willing change in beliefs and attitudes that may result in a change of behaviour [20, 40, 84]. Table 1 lists Cialdini’s principles and how these may be applied to develop persuasive messages for behaviour change.

Table 1. Cialdini’s principles of persuasion and how these may be applied within persuasive strategies to change behaviour [24]
Full size table

An alternative approach that incorporates both the MINDSPACE [35] framework and Cialdini’s [24] principles of persuasion is the Behaviour Change Wheel (BCW) [65]. BCW incorporates the Capability, Opportunity, Behaviour (COM-B) model which is based upon a systematic analysis of 19 frameworks of behaviour change [65]. The COM-B model may be used to perform a “behavioural diagnosis” based upon how the three components of this model interact to form behaviour which also has an effect and impact on these components [67]. BCW may be used to link the findings from this analysis to specific intervention types and policies that support their implementation [65, 67]. In the next Section, we discuss examples of behaviour change techniques within the security domain.

2.2 Behaviour Change for Cybersecurity

As discussed by Briggs et al. [15], protection motivation theory (PMT) [79] has been applied to a range of studies [21, 60, 82] investigating users’ behaviour within a cybersecurity context. In summary, PMT suggests that individuals will perform protective actions based on a prior assessment of a potential threat (threat appraisal) and their ability to engage in recommended preventative measures (response efficacy and coping appraisal) [79].

Nudges have been suggested as a suitable approach towards changing users’ behaviour by aiding decision making related to application privacy settings, in order to avoid unintended disclosure of personal information [1, 2, 9, 10, 30]. Users’ are often willing to accept a trade-off for security and privacy settings due to what has been described as “Psychological distortions” driven by heuristics, cognitive and behavioural biases such as hyperbolic discounting, lack of self control and immediate gain; that may lead to insecure behaviours [1, 2, 29]. Nudges may be applied to address these issues by taking advantage of how users’ may be influenced by such ‘System 1’ and/or automatic cues, to change their behaviour.

For example, Choe et al. investigated positive and negative framing effects via a visual representation of a mobile application’s privacy ratings [23]. Results from this study indicate that this is an effective means for increasing users’ understanding of the potential risks of installing privacy-invasive mobile applications and how this may discourage users to do so [23]. Van Brugeen et al. investigated how messages based on incentives, morality and deterrence may be used for encouraging users to lock their smartphones [94]. Results of this study indicate that messages based on morality are most effective over time, while those based on deterrence are more immediately effective [94]. Nudges incorporated within personal firewall warning messages have also been demonstrated to be effective with increasing users’ risk perception and understanding of the possible negative consequences of their actions in addition to encouraging safer behaviours after receiving such warnings [75].

Kankane et al. conducted a study investigating the effects of five different messages based on incentive, norm, default, salience and ego nudges that may be used to influence users’ password management behaviour [55]. Results indicate that the salience nudge was most effective for reducing participants’ perceived level of comfort with accepting an auto-generated password and the default nudge was the least effective.

Nudges have also been demonstrated to be effective for improving users’ decision making related to selecting wireless network connections. Nudges investigated included using colour coding, order of presentation and a combination of both nudges, to encourage users to select secure rather than less secure wireless network connections. Results indicate that colour coding was more effective compared to ordering, although the combination of both was the most effective for encouraging users to select secure over less secure networks [92].

2.3 Motivation for Study

To develop effective behaviour change interventions to improve cybersecurity, it is necessary to conduct evaluations studies using direct behavioural measurements (actual effectiveness) that provide evidence of how such interventions may change users’ behaviour [38, 95]. The study presented in this paper investigates the actual effectiveness of persuasive messages designed to encourage university staff to complete ISA training. For ISA training to be effective, user participation is essential [3] and lack of motivation amongst users’ to do so may hinder its overall effectiveness [93].

Understanding of actual effectiveness of behavioural interventions calls for repeated laboratory studies (to get insight into ‘efficacy’ with significant control over variables under ideal conditions), repeated field studies (to understand ‘effectiveness’ of interventions where variables are less controlled), and an understanding of the causal mechanisms behind the effectiveness of the intervention (to understand the limits of the transport of results from one field to another) [18, 46]. For our contribution, we conducted one, quite large, field study, focusing on comparisons of a small number of interventions of similar type (explicit persuasive messages) in order to have a reasonable experimental design.

For this study, we had available an existing corporate communications infrastructure, using email, but importantly also access to the underlying organisational structure, for example, the communications team, and sign-off from senior management and the IT department. With the constraints of this real-world context, not all forms and strengths of persuasive message would have been appropriate, or possible, to trial.

3 Methodology

The study was conducted at a university with participants consisting of members of staff only. The university requires staff to complete a range of training courses such as health and safety, equality and diversity and ISA training. The usual procedure for delivering such training involves emailing members of staff a notification that such training is available, required to be completed, and how to access it. Training is usually provided by a web service. Over a period of time, the completion rate for the training is monitored and reminder emails sent to those members of staff who have not yet completed it.

The study procedure for our research followed a similar process, incorporating different types of persuasive messages within notification and reminder emails. We describe each stage of the study procedure in the following sections and an overview of the whole process is presented in Fig. 1.

Fig. 1.
figure 1

Study procedure

Full size image

3.1 Study Procedure

Following current practice at the university, all members of staff received a notification email sent on behalf of a senior member of the management. The email included one of four types of persuasive messages (authority, commitment, reciprocity and unity) which were randomly assigned to participants.

Two weeks after the original notification emails were sent, reminder emails were sent to members of staff who were yet to start the training. Reminder to start (RTS) emails included a slightly modified and reduced version of the type of persuasive message used in the original notification. This was intended to reduce any possible effect of participants feeling manipulated as discussed in related work concerning repeated use of persuasive messages [97].

Four weeks after the original notification emails were sent, all staff who had not yet completed the training received a reminder to complete (RTC) email. This email was sent irrespective of whether any RTS email had been sent and all RTC emails contained the same Social proof strategy. This strategy aims to influence an individual’s behaviour by referring to the behaviour of others in a social context, to encourage an individual to perform the same behaviour [24]. We therefore considered Social proof to be appropriate for RTC emails because this stage of the study provided suitable context for encouraging members of staff to complete the training by referring to those who had already done so.

3.2 Persuasive Messages for Notification and Reminder Emails

Each category of persuasive message used in the study was created using Cialdini’s principles of persuasion [24]. We choose not to use scarcity and liking as we believed these would not be suitable for the application context and goal of persuasion (starting and completing ISA training). Each notification email consisted of three sections and was addressed to each member of staff. The first section included one of the following persuasive messages, a generic motivation statement about the training (regardless of which persuasive message participants received) and instructions on how to access the training:

Authority: The University requires you to complete mandatory Information Security Awareness training. We know that use of our IT systems is crucial to protecting our networks and data.

Commitment: You have been issued with a personal IT account. In using this account, you have agreed to usage conditions including compliance with the University’s Information Security policy. Following this agreement, please complete Information Security Awareness training.

Reciprocity: The University is working hard to protect your personal data and user account against Cyberattacks. To help us with this challenge, we have prepared a short course on Information Security for staff to complete.

Unity: All of us can contribute to maintaining the highest standards of Information Security within our University by completing Information Security Awareness Training.

About the Training: The University is increasingly at risk to a wide range of threats to Information Security. These include unauthorised access to personal data, disruption to the University network and criminal and fraudulent attacks targeting users. To ensure the University is protected against these threats, it is essential that all users are aware of the risks to Information Security and can respond appropriately.

RTS emails consisted of two sections, a slightly reduced and modified persuasive messages of the same category as the prior notification received and access instructions. The RTS strategies are listed as follows:

Authority: The Information Security Awareness training has been available since the <date-prior-email-received>. All members of staff should complete this training to ensure access to the University’s IT Systems is both safe and secure.

Commitment: As part of your agreement with the University regarding safe usage of our IT systems, please start your Information Security Awareness training.

Reciprocity: We want to ensure that the user account we have provided for you to carry out your duties is both protected and secure. Please start your Information Security Awareness training.

Unity: Please start your Information Security and Awareness training and join your fellow colleagues helping to protect and secure our IT Systems.

Staff members who did not complete the training received the following RTC email, regardless of any category of persuasive message previously received:

Social Proof: Please join your fellow colleagues by completing your Information Security Awareness training.

All emails included a standard disclaimer that provided information concerning how data acquired, based on participants’ responses to each email received, would be used for research purposes. Further information included details for how participants could have their data removed from our analysisFootnote 1. A link to the research project website was included, that provided more specific details about the research study, with the exception of how different persuasive messages were being used. This was intended as a means to reduce any possibility of priming participant responses, based on revealing the objectives of the research study. Therefore, at the end of the study period, for all participants who did not request for their data to be withdrawn from our analysis, an email was sent stating that further information about the study was available via the research project website, which stated that a range of different persuasive messages has been used in addition to further information about the objectives of the research study. Participants could still withdraw at this stage.

3.3 Research Question and Hypothesis

The aim of this study was to gather empirical evidence of the actual effectiveness of persuasive messages for encouraging university staff to complete ISA training. As discussed in Sect. 3, the study procedure followed common practice for encouraging university staff to undertake training courses though the use of notification and reminder emails. This provided a means to investigate the actual effectiveness of the persuasive messages by measuring the distribution of participants’ responses throughout the study. Where significant variations in the distribution of participant responses are present, this would suggest that the persuasive messages are not equally effective. This would indicate that at least one persuasive messages was significantly more effective than another. Therefore our research question is as follows:

  • RQ1 Is the distribution of participant responses the same for all persuasive messages?

To develop a testable hypothesis for RQ1, we categorised participants’ responses for those who completed the training as an ordinal dependent variable based on different periods of the study: notification to RTS, RTS to RTC, and RTC to end of study period. A further category for participants who did not complete the training within the study period was also included. This was necessary to ensure the dependent variable was a sufficient measure of actual effectiveness by incorporating all possible participant responses to the persuasive messages during the study. We refer to this measure as response categories. Significant variations present in the distribution of response categories would suggest that at least one persuasive message is more effective than another. Therefore our hypothesis for the study is as follows:

  • \(H^0\) There is no significant variation in the distribution of response categories for all persuasive messages

  • \(H^1\) There is a significant variation in the distribution of response categories for all persuasive messages

3.4 Confounding Variables

Additional participant data acquired for our analysis included gender, gradeFootnote 2 and which school of the university participants were associated with. To ensure participants’ anonymity was preserved, individual grades of participants were banded into three groups. Grades one to four were grouped into a single band as grade one, grades five to seven were grouped into grade two and all remaining higher grades (eight and nine) grouped into grade three. Grade provides an indication of seniority within the university and also corresponds to participants’ age. As such it is possible that participants’ grade may have an impact on the distribution of participant responses during the study. We refer to organisational units of the university as ‘school’ whose disciplines were also anonymised to further ensure participant’s anonymity was preserved. As with grade, it is possible that participants’ responses may vary based upon which school they are associated with. Although we are required to preserve the anonoymity of Schools, we are interested in discovering whether there are any variations in the distribution of participant responses based on School. Due to this study being run in the wild, we could not ensure equal distribution of persuasive messages based on either participant or organisational factors. Therefore our analysis of the results follows the use of non-parametric statistics as these are suitable in cases where the distribution of the dependent variable is not equally distributed amongst the independent variables [83]. In the next section, we report the main findings of this study and an exploratory analysis investigating whether there are any significant variations in the distribution of response categories and participant and organisational factors(gender, grade and school) is presented in Appendix A.

4 Results

The study was conducted with 1592 participantsFootnote 3. The sample included (58%) female, (42%) male, (29%) within grade 1, (52%) within grade 2 and (20%) within grade 3. The distribution of participants across individual Schools is shown in Appendix A, in Table 5, together with results from our exploratory analysis of participant and organisational factors as discussed in Sect. 3.4. We conducted a \(\chi ^2\) test to discover whether there was a significant imbalance of persuasive messages across participant and organisational factors, for each factor. This is considered suitable to discover whether there is a significant difference in the frequencies of two or more independent groups (persuasive messages, participant and organisational factors) [83]. Results from these tests indicate that there is no significant difference between the frequencies of persuasive messages received based on participants gender \((\chi ^2(3) = 5.559, \textit{p} = .013)\), grade \((\chi ^2(6) = 12.591, \textit{p} = .05)\) and school \((\chi ^2(36) = 32.683, \textit{p} = .627)\).

Table 2 list the distribution of response categories for all persuasive messages. To discover whether there was any significant variation in the distribution of response categories for all persuasive messages, we conducted a Kruskal Wallis test, which is suitable for identifying whether there is a significant difference between two or more groups of an independent variable (persuasive messages) which are not equally distributed using an ordinal dependent variable (response categories) [83]. Results from this test indicate that there is a significant difference in the distribution of response categories for all persuasive messages: \((\textit{H}(3) = 8.94, \textit{p} = .03 )\). These results provide support for \(H^1\) by indicating that there is a significant variation in the distribution of response categories for all persuasive messages. Therefore, we address RQ1 by concluding that the distribution of participant responses is not the same for all persuasive messages. This suggests that the persuasive messages are not equally effective and at least one persuasive message is more effective than another.

Following these results, we conducted a post hoc Dunn’s test, to discover whether there were any specific significant variations in response categories between the persuasive messages using a Bonferroni correction to control for type 1 errors [37]. We discovered a significant variation in the distribution of response categories between the Unity and Authority persuasive messages \((\textit{p} = .03, \textit{r} = .1)\). Participants who received the unity persuasive message completed the training earlier, with fewer not completing the training compared to those who received authority. No other significant variation in the distribution of response categories was discovered for any other pairwise comparison of persuasive messages. Therefore, we conclude that the unity persuasive message was more effective compared to the authority persuasive message only. We note that despite a significant variation in the distribution of response categories between the unity and authority persuasive message, the effect size is small [27].

Table 2. Distribution of response categories for each persuasive message
Full size table

5 Study Limitations

As part of the conditions for ethical approval to perform this study, it was necessary to acquire informed consent by participants using a disclaimer included within the emails sent to participants during the study. Consequently, it is possible that participants may have responded differently if they were not informed about the study in progress. To minimise this effect, participants were only informed that a study on the use of persuasive messages was being conducted but not that different message strategies had been used throughout. This information was later released on the project website and participants who did not opt out of the study, regardless of whether they completed the training or when, received this information as part of the thank you email.

Another condition as part of our ethical approval included the need to mention within the disclaimer, that regardless of whether participants choose to opt out of the study or not, that it was mandatory for members of staff to undertake training as part of university policy. This may also have influenced the participants in addition to type of persuasive message received and as such, each email contained some aspect of the authority principal.

The live nature and environment for this field study limited the way in which the study could be run, and this limits the conclusions that can be drawn. For example, no control or neutral (no persuasive message applied) condition could be applied to participants. This was necessary as part of the conditions for ethical approval to perform the study and to fulfil the university requirement that all members of staff complete the training. It was considered that participants who did not receive a persuasive message containing at least one persuasive principle may be less likely to complete the training compared to those who did receive a persuasive message containing at least one persuasive principle. For a second example, while messages were intended to emphasise one persuasive principle or other, they (through the message itself or source and channel) often contained other factors that could influence behaviour (e.g. other principles). For example, all messages were known to come from an authoritative source (the university). We therefore cannot conclude in all certainty that the effectiveness of a message was a result of its privileged persuasive principle, rather than the result of some other factor.

6 Conclusions and Discussion

This paper presents a study of the relative effectiveness of four persuasive messages for encouraging users to complete ISA training. This study is one of very few which was performed in the wild and measured the actual effectiveness rather than perceived effectiveness of persuasive messages. We observed that there was a significant variation in participants responses to the persuasive messages. This suggests that some persuasive messages differ in effectiveness. There was a significant difference between the responses to the unity and authority persuasive messages, but the effect size associated with the significant variation in participant responses between the two was small. This is perhaps because the different messages had mild variations of emphasis of different persuasive principles, rather than using completely different principles. As discussed in Sect. 3.2, only the first section of each email contained one of the persuasive messages with the remaining content being identical for all emails. Furthermore, as discussed in Sect. 5, each email included some aspect of the authority principle within the disclaimer which may have influenced participants. However, this means that such small changes of emphasis may not make a practical difference.

Our results concerning the unity persuasive message would appear to support claims that individuals may alter their behaviours (within the context of cybersecurity related behaviours) to match others whom they identify as being a part of the same group (in the case of our study, members of staff at the University) [11]. It is possible that the unity persuasive message triggers social capital as the motivation for participants to complete their training, through this message’s emphasis on shared collaboration towards a common beneficial goal. As discussed by Sasse et al. [58] individuals within an organisation are, to a certain degree, “emotionally attached” to the organisations they are apart of [61, 80] and may be motivated and capable of performing protective behaviours [14, 56, 57], which is the overall objective for engaging with ISA training. Herath et al. suggests that motivation to perform security related behaviours (in the case of our study engaging with ISA training) may be influenced by users’ “closeness” to organisation they are a part of [50]. At the same time it is possible that the authority message constrains and/or weakens social capital as a motivator by implying that although completing the training is important, this is nevertheless a mandatory (enforced) request. Further studies are required to clarify this.

In future work, we plan to investigate the perceived effectiveness of the persuasive messages using a scenario based approach that provides a greater means to measure specific individual and organisational factors, compared to a field study in the wild. We intend to discover whether the results from such a study would yield similar results with respect to the variations in participant responses to the unity and authority message and to what extent more specific measures of participant and organisational factors may influence participants’ susceptibility to the persuasive messages.