Keywords

1 Introduction

ATMs (Automatic Teller Machine) are a type of SST (Self-Service Technologies), and the ATM domain considers ATMs that banking customers utilize for doing transactions related generally to cash. There are still many people who use ATMs to withdraw cash [1], and financial institutions need to deliver for their customers an acceptable UX (User Experience) in all its channels. It is necessary to provide methods and tools that let UX and Development teams do improvement processes, evaluations, and assessments. These methods and tools should be both qualitative and quantitative. Among quantitative methods, UX Metrics are currently very useful in this context of UX improvement.

This paper is the continuation of a previous work conducted in the past [2, 3], where we found the close relation that exists between Usability and Security when someone is using an ATM [4] and also evidenced the lack of specific metrics for these two aspects in the ATM domain. According to this, covering the lack of specific metrics for the ATM domain is very important and necessary for the industry. With a set of metrics, it will continue to contribute to the currently lacking information on the knowledge of the application of usability in ATM interface designs, which causes severe problems when users interact with this self-service [5].

In this paper, the authors explain the methodology used to search and select Security and Usability metrics used by other banking domains, such as Internet Banking. Another input was the metrics that the ISO 25000 standard provided. With that input, we developed a proposal of a set of Usability and Security Metrics for the ATM domain, having an expert judgment for each of the 35 metrics. To end this work, the set of metrics was validated by conducting user tests and then comparing the results obtained using questionnaires SUS (System Usability Scale) [6] and the results obtained by applying the proposed metrics.

In the second part, this paper contents an explanation of the methodology that we used to collect the guidelines and select the input metrics. In the third part, we present the proposal for the usability and the security interface metrics. In addition, we explain the results from the judgment expert. In the fourth part, we explain each part of the process to validate the proposal metrics: Selecting Interfaces, Template development, user test, and Interface evaluation. At the end of the fourth part, we present the results of the validation process. To finish, in the fifth part, we discuss the conclusions and the possible future work related to this paper.

2 Methodology

In this part of the paper, we explain the input of this work, the ATM guidelines, and the metrics that we used to build the new metrics.

2.1 Research

As mentioned, we already have a list of usability and security metrics for Internet Banking channel [2]. This list of metrics has 13 metric categories divided into two parts: 6 of them are metrics to evaluate the security of that channel, and 7 of them are metrics to evaluate the usability of that channel’s interfaces.

In ISO 25000 [7], we found six groups of usability metrics that have for evaluating the degree to which a user can use a product or system to achieve specific objectives with effectiveness, efficiency, and satisfaction in a specific context of use. For security metrics, five groups are mentioned to assess the degree to which a product or system protects information and data; so that people or systems have the degree of access to the data appropriate to their types and levels of authorization. Each of the metrics in the ISO25000 has a metric ID. In Table 1, we informed which Metrics ID were considered for this work, specifying which group and type of metric they correspond. As a result of searching for usability and security metrics in the literature and ISO25000, we obtained 193 metrics.

Table 1. Groups of metrics from ISO 25000
Full size table

2.2 Selection

In order to identify which metrics, of the 193 metrics found, make sense and can be adapted to the ATM channel, we worked by reviewing each of the metrics and looking for if it was possible to associate it with one of the guidelines proposed for the ATM domain [3, 8]. It is necessary to specify that the compiled guidelines are based on the need to convey confidence and clarity to users when interacting with ATMs [9]. Table 2 and Table 3 show which metrics were associated with each guideline, and we informed which was the input if that metric was taken from the ISO 25000 or from the metrics found in the Systematic Literature Review (SLR).

Table 2. Association between usability guidelines and metrics
Full size table
Table 3. Association between security guidelines and metrics
Full size table

3 Proposal of Metrics for ATM Interfaces

Following the definition of metrics as a measurement scale and method used for the measurement of attributes that influence one or more quality characteristics [10] and after making the association between the 25 ATM guidelines and the metrics found, we proceeded to adapt these metrics to the ATM domain, adapting the language used or the terms that were specific to Internet Banking. Besides, new formulas were generated for the metrics without a formula, and the existing formulas in ISO 25000 were refined.

The result was a proposal of 35 metrics for ATMs, divided into two groups, the first one, with 23 metrics focused on quantifying aspects of Usability, and the second one, with 12 metrics focused on measuring aspects of security in the interfaces. Each proposed metric includes a metric identification code, the metric’s name, a description of the information that the metric provides, and the formula to calculate.

It is essential to highlight that in all the proposed formulas, the recommended value is the closest to number 1.

3.1 Proposal of Usability Metrics for ATM Interfaces

Table 4 shows the 23 metrics that correspond to the Usability Metrics for ATM interfaces.

Table 4. Proposal of usability metrics
Full size table

3.2 Proposal of Security Metrics for ATM Interfaces

Table 5 shows the 12 metrics that correspond to the Security Metrics for ATM interfaces.

Table 5. Proposal of Security Metrics
Full size table

3.3 Expert Judgment on Proposed Metrics

The 35 proposed metrics were adapted to a survey format, in which each participant was asked to rate, on a Likert scale [11], the following aspects:

  • Clarity of the metric

  • Identification of characteristic being measured

  • Applicability of the metric

  • Appropriate formula

Confirming that the proposed metrics comply with these four aspects will help to decide if it is a valid metric according to the above, referring to what was developed by Kitchenham [12]. It was established that those with a score greater than three would be taken as accepted metrics, while the metrics that obtained a value equal to or less than three would be discarded from the list of proposed metrics.

Four domain experts from two of the major national banks and two ATM interface design experts were contacted to respond to the survey provided. Table 6 and Table 7 show the average result obtained in each aspect for each of the metrics.

The results of the surveys showed that the 35 metrics obtained an average score greater than three in the four aspects mentioned. As mentioned, the score would be the deciding factor to exclude any of the proposed metrics. Since none of the metrics obtained a score equal to three or less, we concluded that the 35 metrics developed are valid to carry out an evaluation of usability and security of interfaces for ATM.

Table 6. The average score for Usability metrics
Full size table
Table 7. The average score for Security metrics
Full size table

4 Validation of the Metrics

In order to validate the proposed metrics, two evaluations were carried out for each transaction flow that exists in the ATMs of banks that operate in Peru. The first evaluation was carried out with the proposed metrics, and the second evaluation was carried out using the SUS questionnaire. For these evaluations, it was necessary to perform a user test [13] to complete the SUS questionnaire and collect the information necessary to complete all proposed metrics’ evaluation fields.

4.1 Validation Process

Selecting Interfaces.

For this validation, the four most important banks in Peru were selected [14], which we will call A, B, C, and D. Subsequently, it was selected to evaluate the transaction to withdraw money in local currency from a savings account with a debit card since this transaction is the most used in ATMs.

Template Development.

An Excel template was prepared to facilitate the calculation for the evaluators. This template had the following sections:

  • Instructions: This section mentioned the way in which the fields in the Usability Metrics and Security Metrics sections should be filled.

  • Definitions: This section gave some definitions of terms used in metrics to clarify any doubts.

  • Usability metrics: The 23 metrics were presented with the spaces to fill in the answers to the questions posed in each of them. This sheet indicated with an asterisk which were the questions that needed information from the user test.

  • Security metrics: The 12 metrics were presented with the spaces to fill in the answers to the questions posed in each of them. This sheet indicated with an asterisk, which were the questions that needed information from the user test.

  • Results: By having formulas related to the previous tabs, this section showed the results of each of the metrics and the result obtained by taking an average of all the metrics.

User Test.

A user test was carried out to collect that information to complete the fields marked with an asterisk. The metrics for which it is necessary to perform a user test are:

  • U4

  • U10

  • U12

  • U15

  • U17

  • U19

  • S7

  • S10

The user test was conducted with 20 users aged between 22 to 71 years, all with experience making ATM withdrawals. This information was obtained from the pre-test that was carried out. Furthermore, all participants were informed of the objective of the test.

For the test, a prototype of the withdrawal flow of an account in Soles of the 4 main banks in Peru was made. These prototypes were made with the Invision tool [15] and placed in real ATMs located in a development laboratory (see Fig. 1).

Users performed the following tasks in random order:

  • Bank A: Withdraw S/20 from a savings account and request to see the balance on the screen.

  • Bank B: Withdraw S/20 from a savings account and do not request a voucher.

  • Bank C: Withdraw S/50 from a savings account and do not request a voucher.

  • Bank D: Withdraw S/20 from a savings account and request to see the balance on the screen.

In the end, the participants completed a SUS questionnaire for each flow performed.

In addition, we asked some questions related to the mentioned metrics, taking as support the screens of the printed prototypes.

Fig. 1.
figure 1

Users performing the test

Full size image

Interface Evaluation.

To perform the evaluation with the metrics and obtain the data that serves to answer the questions of each metric, multiple withdrawals were carried out in the four banks selected to have the necessary data. This activity and the analysis of each screen of the withdrawal flow allowed completing all the necessary fields for the 35 metrics.

Each of the evaluations was carried out in a different template to avoid confusion and handle the data separately.

4.2 Results

As a first step, we collected the results of the SUS questionnaires. To obtain the final score, which ranges from 0 to 100, we add the results with the following formula [16]:

$$ \begin{aligned} {\text{S}}\ =\ & \left[ {\left( {{\text{PP1}} - {\text{1}}} \right){\text{ }} + \left( {{\text{PP3}} - {\text{1}}} \right){\text{ }} + \left( {{\text{PP5}} - {\text{1}}} \right){\text{ }} + \left( {{\text{PP7}} - {\text{1}}} \right){\text{ }} + \left( {{\text{PP9}} - {\text{1}}} \right)} \right] \\ & + \,\left[ {\left( {{\text{5}} - {\text{PP2}}} \right){\text{ }} + \left( {{\text{5}} - {\text{PP4}}} \right){\text{ }} + \left( {{\text{5}} - {\text{PP6}}} \right){\text{ }} + \left( {{\text{5}} - {\text{PP8}}} \right){\text{ }} + \left( {{\text{5}} - {\text{PP1}}0} \right)} \right] \\ \end{aligned} $$

S = Sum

PP = Question score

Then, we multiplied 2.5, and the result obtained (S). The average of the results obtained was calculated, and the results are shown in Table 8.

Table 8. SUS finale score
Full size table

As a second step, the results obtained with the metrics proposed in each bank were reviewed. The result obtained by each of the banks, according to the calculations of the metric templates, is shown in Table 9.

Table 9. ATM metrics finale score
Full size table

The evaluation carried out using the metrics proposed in this work given as a final score, a result directly related to that obtained in the evaluation with the SUS questionnaire. Figure 2, shows how the four banks obtained the same position compared to their competitors.

With this result, the validation of the proposed metrics was approved.

Fig. 2.
figure 2

Results comparison

Full size image

5 Conclusion and Future Works

This work responds to the problem of having subjective evaluations of ATM interfaces since a tool is proposed to carry out quantitative evaluations.

The contribution of the guidelines raised above and ISO standards was considered essential since the metrics have been raised in correlation to the established guidelines for the usability and security of ATM interfaces.

It was validated that the 35 metrics satisfy the needs expressed by the people who work related to the design and development of ATM interfaces.

In addition, it was determined that, in the face of an improvement in issues of usability or security in the interfaces, no matter how minimal, it will be evidenced in the score of the result.

As future work, it is considered necessary to continue carrying out evaluations with the elaborated metrics, carrying out user tests with other or new operations, and confirming the results obtained and strengthening the tool.

It is also proposed to automate or facilitate data entry to the templates to help the evaluator obtain results more quickly.