Keywords

1 Introduction

The simplicity and assumed power of the technology acceptance model (TAM) [1, 2] accounts for its continued popularity [3]. In its simplest form, the model claims that Perceived Ease of Use (PEOU) predicts a potential user’s Attitude Towards using, which in turn leads to the Behavioral Intention to adopt the technology. This effect is often mediated by Perceived Usefulness (PU). At the same time, there is increasing evidence that the model is not able to deliver against its potential [4, 5]. Later iterations [6, 7] confound its original elegance with contextual factors (job relevance, voluntariness, etc.) and adopter characteristics (gender, self-efficacy, etc.). It is unclear whether such extensions simply make explicit the external variables Davis et al. identified [1] or underplay more significant failings [8, 9]. TAM starts to abandon simple measures of PEOU and PU, introducing contextual factors mirrored in its competitor, the Diffusion of Innovation theory (DOI) [3, 10]. It may even undermine significant issues in technology design [11]. Human and social factors need to be given greater prominence [12], especially affect [13]. Indeed, technology is not simply a passive component, but may have an important and equal role in a complex human-machine network [14].

1.1 Investigating Conditions of Technology Acceptance

Although ostensibly easily accessible, PEOU may not be as instrumental in determining technology acceptance and adoption as once thought. We previously reported an exploratory study questioning the dominance of PEOU [15], operationalized via the System Usability Scale (SUS) [16]. Adopting a mixed-methods approach, we identified conflicting results for two cohorts working in healthcare in Italy and in Spain. In both cases, SUS scores were below threshold, suggesting that the technology was not easy to use and therefore in line with TAM predicting that it would not be acceptable. This was the case for a cohort of ICT professionals in Italy. However, a mixed group of ICT users and managers in the Spanish study working collaboratively were more responsive to the technology, developing a task-focused narrative where specific scenarios might enhance and support them in their day-to-day responsibilities. We speculated at the time that the effect may be due to priming (i.e., usability) or the collaborative conditions under which the Spanish trial but not the Italian one was run (engaging with one another to use the technology encouraged a more rounded view of its PU). We need to consider, though, whether different effects between the Italian and Spanish cohorts may be the result of some other artefact. In this paper, therefore, the goal is to validate the reproducibility and generalizability of our previous findings. That being the case, we also consider how to take this work further.

2 Method

As a baseline technology, we used a security modelling tool which finds risks and offers mitigation strategies in cyber physical systems [17, 18]. The technology was chosen initially since organizations often struggle to identify exposures to risk, as do human agents when faced with complex ICT systems [19]. From the earlier study [15], we predict that PU is not dependent on PEOU, provided that responses are elicited via appropriate methods to allow participants to respond more freely.

2.1 Research Questions

To investigate further the issue of technology acceptance, we focus in this paper on two questions:

  • RQ1: Do we see the same effects emerge consistently?

  • RQ2: Do we see similar effects with a different cohort on a related task?

These are intended to address concerns about reproducibility (RQ1) and generalizability (RQ2).

2.2 Design

There were three studies: one in Italy, one in Spain and one in the UK, each with two iterations making a total of six sub studiesFootnote 1. These studies were carried out originally as technology validation trials, with the initial round intended as formative and the second and final round as summative. They had not been specifically designed to explore user acceptance or adoption, and so were not planned or run with the types of controls associated with much empirical work. However, we claim greater ecological validity in that these were effectively field observations. For the Italian and Spanish trials, the technology was used as a standalone, visual design tool intended to promote understanding of cybersecurity within a healthcare environment. In the UK, by contrast, the base technology was integrated into an online social care service.

2.3 Participants

For the first iteration of Italian and Spanish trials, two small cohorts of self-selecting IT professionals were recruited, 5 in Italy and 4 in Spain. All participants had up to 20-years’ experience representing different IT disciplines. Five Italian engineers and developers from the hospital’s IT department were recruited in the Ospedale San Raffaele (OSR) in Milan, including Application Development & Management; Service Desk; Privacy, Procurement & Control; CRM, Business Intelligence & Process; and Enabling Services & Security. Four self-selecting participants from the Biocruces Bizkaia Health Research Institute in Spain including a Bioinformatics Technician, a Computer Science Engineer, a Software Developer and a Database Manager. In the second round, two participants in each country returned (a software developer and system administrator in Italy; and the Bioinformatics Technician and Database Manager for the first round in Spain). In the UK, 8 participants working for a social care organization took part; roughly half were IT professionals and the others volunteers. In the second, 14 participants split between IT, Charities and management roles took part.

2.4 Data Collection

For the Italian and Spanish trials using the technology as a standalone visual design tool, participants were given a target scenario involving the secure country-to-country transfer of healthcare data. The security of the data is, of course, paramount for both legal [20] and ethical reasons [21]. Using the visual design tool, participants were asked to draw up an appropriate cyber technical system and to comment how realistic the risks and mitigation strategies were that had been automatically generated by the tool. In the first round, Italian participants worked in pairs, whereas the Spanish participants all worked as a group. In the second iteration, the two participants at each site worked together. The two co-authors (MNJ and BLM respectively) supported them and made notes during subsequent discussions when participants were encouraged to “think aloud” about their experience of using the technology.

For the UK trials, the domain was switched to social care. Here, there is still the legislative responsibility to protect personal data. However, the focus now is on effective delivery of social care. The technology was therefore integrated with an existing online social care service to handle privacy and data security. Participants were asked to role-play users as they carried out typical tasks with the service. They were then asked to fill in a privacy and a usability questionnaire. The co-author (RB) facilitated both sessions and provided support if needed. She also conducted semi structured interviews with participants about their response to the service as it handled the tasks they were attempting to complete.

2.5 Data Analysis

We adopted a mixed-methods design to re-analyze the original results which used quantitative instruments and semi-structured interviews. Using both qualitative and quantitative methods has become increasingly common certainly within healthcare [22]. For the first iterations in Italy and Spain, the SUS was used to estimate PEOU [15, 16], providing an overall usability score with scores below 68 regarded as poor. In the UK, by contrast, a locally generated instrument was used based on the company’s experience with technology trials and rollout. Scores were reported in relation to the Likert label they most closely approached: for instance, a score of 4 as “Agree”, and of 5 as “Strongly Agree”.

For the semi-structured interviews, thematic analysis [23] was used across all sites and iterations, focusing on two main themes: PEOU and PU. The notes the researchers took to record what participants said were first analyzed, therefore, to identify the main themes. Subsequently valence (positive versus negative) was assigned to each of the utterances associated with the themes.

3 Analysis

As previously reported [15], during the first iteration neither the Italian nor the Spanish participants on average rated PEOU, operationalized via the SUS, above the standard threshold, a score of 68, with some individual scores a lot higher and some lower. Comments from the Italian participants confirmed PU to be poor. By contrast, the Spanish cohort were more positive and began to describe potential benefits to the organization for identifying risk and introducing consistency. The first question then (RQ1) was to establish whether this trend was repeated for the second iteration.

3.1 Investigating Reproducibility

We validated the reproducibility of the findings with results from a second iteration of testing with the same participants and an updated, improved version of the technology. During this iteration, PEOU was not measured explicitly, focus was placed instead on encouraging participants to describe their response to the technology. Table 1 summarizes comments related to PEOU and PU by valence for both Italian and Spanish participants together. For instance, there were 6 negative comments on PEOU, but 11 positive comments on PU. The Italian participants did criticize some aspects of the user interface (UI) and tool (performance and visual presentation) and felt the technology required prior experience (3 negative comments on PEOU).

Table 1. Summary of positive, neutral and negative comments from Italian and Spanish participants.
Full size table

However, both the Italian and Spanish participants commented on the potential for process improvement: the tool would effectively take an essentially manual (“whiteboard”) one currently to a much more efficient semi-automated one (9 positive comments about PU).

Italian User 2::

Because the same kind of process we are doing but by using whiteboard, in mind or talking with the people. When we have to do a project and we build an infrastructure, usually we are not aware of all the threats and possible solutions that are available. I think this tool can help us a lot in finding threats and solutions.

What is more, understanding the concept underlying the technology, the other participant starts to think beyond the immediate context of the validation exercise and the Perceived Usefulness of the design tool for their own needs:

Italian User 1::

If my job was this, I think that probably it would help me a lot.

And then subsequently:

Italian User 1::

The technology itself is not that difficult to understand but for me it will be difficult to think what structure I have to design just because it is not my job.

This participant goes on later to criticize specific features of the UI: the search function, the color palate and the lack of a zoom function in the design canvas. Yet, when asked whether the technology would help those whose responsibility is to design, implement and run secure infrastructures, the same user simply observes: “Yes, I think so”. So, the user is aware of specific shortcomings in the UI, and yet believes overall the technology is easy to use. More importantly, although not applicable to their own job role, they start to make sense of the technology when considering what other colleagues might need and how they might benefit from using it.

The Spanish users also note issues with performance and some difficulty with descriptive labels in the UI (3 negative comments on PEOU). However, they also highlighted the potential usefulness of the technology for a range of colleagues (architects, engineers and implementers) to visualize and understand issues associated with a complex infrastructure (2 positive comments on PU).

Spanish User 2::

Building a technology infrastructure [with this tool] has recently attained widespread attention in the Engineering of the structure of a system. It helps architects, engineers and constructors to visualize what is to be built in simulated environment and to identify potential threats.

Both Italian and Spanish participants, therefore, explore the potential usefulness for their respective organizations. Even if they can see no direct benefit to themselves, they are still able to appreciate how the technology might fit for others. Without the priming task of SUS, they seem to downplay PEOU, focusing instead on the umbrella issues of the security of health data and potential utility of the technology. One of the Italian participants even suggests that performance issues would be acceptable given the advantages available.

Italian User 2::

I think doing this task in real life is so time consuming so the tool is somehow justified to take this much time.

Accommodating technology shortcomings on account of potential benefit has been attested elsewhere [24, 25]. However, the experience of these limited validation trials suggests that potential adopters are willing to explore potential rather than simply reject on the basis of poor usability.

The earlier results [15] are therefore confirmed in terms of reproducibility. Despite the small number of participants, they were able to identify how the technology might fit within their organization to improve existing processes even if not directly relevant to them. They are still aware, of course, that the technology under test has some way to go to improve usability. However, this has not distracted them from seeing potential. Indeed, they have even been willing to think of ways that technical shortcomings might be tolerated if not completely overlooked.

3.2 Investigating Generalizability

To address questions of generalizability, we revisited the results of a previous validation test, based on the same underlying technology trialed for social care in the UK. This time, the technology had been embedded within an online social care service. Results were available for two iterations with overlapping, though non-identical participants, 8 in the first round and 14 in the second. Instead of the SUS, participants were asked to rate both their attitude to privacy [26, 27] and technology usability. Participants reported broad agreement on a 5-point Likert scale for both usability and usefulness, and for each iteration, with 1 = Strongly Disagree, and 5 = Strongly Agree. For the first cohort, usability was ranked at 4.33 (median score) and usefulness at 4.00; these correspond to a judgement of “Agree”. For the second iteration, the median scores were 3.92 (usability) and 4.00 (usefulness). Wilcoxon signed-rank tests showed that there were no significant differences between usability versus usefulness for either iteration (Z = −1.378, p > 0.05 for the first, and Z = −0.120, p > 0.05 for the second). In both cases, therefore, usability and usefulness were rated equally well. If PEOU really is a predictor for intention to adopt, mediated by PU, then we would expect user attitude to be broadly positive when discussing the technology.

A qualitative analysis of outputs from semi-structured interviews is summarized in Table 2. For each of the two iterations (first and second), the valence of attitudes to PEOU and PU is markedly negative overall. There is a greater willingness in the first iteration of testing to explore usefulness (16 versus 15) than in the second (28 versus 52). Yet, social care participants do see potential benefits, not least for their own interactions with automated services:

Table 2. Summary of positive, neutral and negative comments from English participants.
Full size table
P4 (1st iteration)::

I would consider using this in daily life

P2 (2nd iteration)::

I trust that it works, its transparent and makes me feel secure

P10 (2nd iteration)::

The platform is a sensible idea – when is it coming out!

P11 (2nd iteration)::

Most people don’t realise what they have signed up to, and tick a terms box without reading it – this should wake them up to realising the consequences

These sorts of comment suggest that the technology performs a specific job, namely provide the client (the data subject) with easy-to-use control of their own privacy. It informs the preferences they might want to make without forcing them to run through all the possible consequences: this is the type of empowerment that legislation was intended for and which users (data subjects) may simply be unable to understand and exploit [28].

But on the other, just as the Spanish and Italian participants thought of how the technology might help their colleagues, so the UK social care workers are concerned for their service users – who may be vulnerable or simply not have the experience with technology – and how they would react to such a system elicits a different and more critical response:

P4 (1st iteration)::

if my mum were to use it she would need a walkthrough or a guide. For instance like the [ABC] sign up process is a guided journey, [the service application needs] to be more guided

P8 (1st iteration)::

I’m not confident that someone less tech-savvy (e.g. an ABC client) would [find it easy to use]

P6 (2nd iteration)::

“intuitive web-based dashboard” language would be difficult to understand for some people

This ambivalence in reaction was not reflected, of course, in the quantitative metrics where usability and usefulness were equally and positively rated. It may well be that participants responded to the survey questions from their own perspective rather than thinking about their clients. Those metrics would predict a willingness to explore the potential of the technology, which is found in what they say. Yet their comments on usefulness and usability include their clients for the most part. This may reflect the fact that the validation protocol required them to perform a series of typical tasks that the service users (their clients) might encounter. So, as far as usability and usefulness is concerned, they would typically focus only on how it affects the participants in achieving the tasks they’d been set. However, there is an added consideration with usability specifically with the social care services they provide for their clients in the UK: those validating the platform with its enhanced security settings have a responsibility not only to see appropriate controls in place to protect personal and sensitive data, but they also have a responsibility to their clients and how they would get on with the technology.

As far as generalizability is concerned, this second set of results in a different setting (UK social care) suggests that there is indeed no clear-cut relationship between PEOU and PU in terms of adopter intention. Others have also concluded that a direct causal relationship is not as robust as often claimed [29, 30]. But there is evidence too that any such relationship is at best ambiguous in multiple contexts and across different participants. Where those participants are encouraged to explore, their responses to the technology under test and not simply score it via quantitative survey questions focus on making sense of the technology in the contexts they see as relevant.

4 Discussion

A post hoc survey designed to capture a user’s perceptions of technology usability is problematic for several reasons. The SUS and similar instruments rely on the user’s memory of their experience rather than their experience itself. Further, assuming that the user is actually reporting their own experience rather than their expectations of how someone else might view the technology may not be justified. Additionally, of course, if they are experienced users, they may be more critical of less significant features or even biased towards others which they would not implement themselves. The different iterations of the original TAM by Venkatesh and his colleagues [7, 8] start to introduce mediating factors such as experience, willingness and self-efficacy. Even Davis had originally allowed for external variables [1] influencing Perceived Ease-of-Use and Perceived Usefulness but without attempting any systematic investigation. Thatcher and his colleagues are more explicit and position technology adoption within a broader socio-technical context beyond Perceived Ease-of-Use and Perceived Usefulness [31]. It should be no surprise, therefore, to find ambivalent indications of ease-of-use responses coupled with more comprehensive verbal evaluation of the possibilities afforded by the technology.

The main characteristic that all participants across iterations and sites share is their existing professional association with technology, as users, developers or service providers. Only the (UI) developers are likely to be concerned directly with the look and feel of the technology. The first Italian participant in the second iteration demonstrates this with their very specific comments about the UI, but free form responses identifying potential usefulness of the technology (see above). Instead, the focus for the participants here is on their professional responsibilities. Referring to DOI theory [10] and extrapolating from the documented TAM case for clinicians [29, 32] where organizational context and personal belief systems have an effect, these users are contractually and morally obliged to deliver secure, privacy enhancing services.

Adding to the technology characteristics operationalized initially via PEOU and PU (TAM) and perceived advantage, complexity and ease of integration (DOI), Hackman & Oldham’s formulation of the Job Characteristics Model (JCM) [33] and Karasesk’s Job Demand Control Model (JDC) [34] provide additional insight in terms of job characteristics and psychological state. In JCM job characteristic Task significance is high (i.e., the privacy and security of user data) with punitive fines for data protection breaches [20] and a loss of data subject trust [35]. This leads to High job strain (JDC) and therefore increased psychological stress. We believe that there is evidence in our re-analysis of participant responses to the technology under test of a realization of the potential for such technology to handle the Task significance amid public concern and regulatory change. In so doing, this reduces the risk of psychological strain. This in turn enhances the experienced meaningfulness and knowledge of results [33]. According to the original JCM, such outcomes would increase job satisfaction, performance and motivation.

In psychological terms not only does the individuals regain control (JCM) and reduce sources of stress (JDC), but shifts their focus away from the burden of regulatory and operational responsibility towards the driving force for their doing their job. Their motivation has shifted from extrinsic constraints and risk to the intrinsic realization of increased personal autonomy and feelings of competence [36].

5 Limitations and Future Work

Repurposing secondary data by reworking the results from previous studies may be criticized from an empirical methodology perspective. Participants in the original studies had different expectations from the experimental setup, and no appropriate controls to reduce the potential and unwanted influence of confounding factors. It may be argued, for instance, that the participants were sensitive to the demand characteristics of the situation: they are aware that the facilitator represents the technology and so may be more measured in their judgements. Opportunity sampling may conversely have reduced the potential for independent evaluation: these are technology users already and so perhaps compare the technology under test with what they know.

Equally, we would argue that the re-interpretation of the language they use to describe their experience provides a richer perspective on how they view the technology. These informal responses of potential adopters correspond more closely to a DOI approach: we are not simply looking at technology features and how they might influence adoption, but also the ability of potential users to see how the technology fits, and how they communicate their perceptions to others [10]. So, if language is the basis of the social construction of meaning [37] then it affords a more rounded understanding of how technology is likely to fulfill the needs of potential adopters. Exploring how individuals make sense of experience [38], creating a task-oriented progressive narrative which suits their own needs [39], we intend to explore technology acceptance in a follow-on, primary data study eliciting responses to the technology used in the studies reported here as it is seen to fulfil the practical needs of a controlled and cohesive group of potential adopters.

6 Conclusion

The Technology Acceptance Model predicts a strong causal relationship between Perceived Ease of Use and Perceived Usefulness. The claim is that if users find a technology easy to use then they would see its usability. However, revisiting validation tests across cohorts of IT and social care professionals in health and social care has called into question this assumed causal relationship. Allowing users to engage with a technology in context seems to encourage a more pragmatic perspective. Once potential adopters begin to make sense of the target technology as it addresses their needs, they are willing to accept shortcomings in its implementation. Planned research will in future explore the spontaneous use of narrative as a methodology to identify the intention to adopt.