Keywords

1 Introduction and Research Question

In the context of this paper, the application of usability testing for users with cognitive impairments was investigated. There are a variety of established methods, but most of them do not consider impaired people. According to the Federal Statistical Office, around 7.9 million severely impaired people live in Germany. This results in a share of 9.5% of the total population. These people often have physical impairments and/or cerebral disorders. Around 13% of the severely impaired people are cognitively impaired, which results in a number of 1.03 million people (see Fig. 1) [1, 2]. Their mental or emotional health is not in balance and must be considered individually. These statistics refer to Germany only, as there are country-specific classification and rating systems for impairments and cognitive impairment. For example in the development of websites or applications, these physical and cognitive aspects have a variety of effects regarding to the requirements in information design. Too little attention is paid to them in order to ensure accessibility of digital information. Not only cognitively impaired people would experience a great advantage from appropriate consideration. The adaptations would mostly also be helpful for other user groups e.g. children or elderly people. There are already guidelines, such as the Web Content Accessibility Guidelines [3], which are an international standard for the accessible design of websites and other web contents. Appropriate websites are thus accessible to people with sensory and motor (and to some extent mental) impairments, e.g. they can grasp the digital information offered and make necessary inputs. User-centered development also takes into account the requirements and abilities of the intended target group. By involving the target group, for example in requirements analyses and usability tests, the individual abilities of end users are recognized and taken into account in the development of websites and applications.

Fig. 1.
figure 1

Types of impairment 2019 Germany [2]

Full size image

The empirical evaluation of two applications has raised the question of feasible testing methods. It is necessary to investigate how usability testing methods need and can be adapted accordingly to the linguistic, emotional, and cognitive abilities of users with cognitive impairments. Due to COVID-19, part of the tests were conducted as remote tests. Therefore, the possibilities and limitations of remote testing with the target group were also examined in more detail, as this form of testing has gained relevance in times of COVID-19. Therefore, this paper deals with the two research questions:

How must usability testing be modified for cognitively impaired persons?

What are the advantages and disadvantages of on site and remote usability testing for cognitively impaired persons?

The aim of the work is to define and test the modification of usability testing for persons with cognitive impairments. In addition, the application of on site and remote usability testing is compared. It will become evident that it is quite possible to test with cognitive impaired people. Adjustments have to be made to the concept and process of testing. This will be discussed in detail in the course of the paper.

2 Related Work

On the basis of the foregoing explanations, the target group of cognitively impaired persons is first examined in more detail as an important component of testing. There are various procedures as a tool for assessing the competencies of cognitively impaired people. They lead to a better understanding of the user group of cognitively impaired people and are summarized below. In addition, existing work such as studies and guidelines for the use of usability testing with cognitively impaired people will be examined and related to the procedure and aspects of usability testing.

2.1 Analysis of Procedures for the Assessment of People with Impairments

The well-founded definition of the target group is essential for the application of usability testing. Cognitive impairment is defined as a collective term for impairments in external and internal information processing in the brain. Briefly summarized, a cognitive disorder has an impact on a person's thinking, which mainly affects the functions of perception, attention, memory, action planning, judgment, problem solving, and communication [4]. Experts in the healthcare field use various procedures as a tool to assess individuals with cognitive impairment. For a better understanding a short selection is summarized in Table 1. Basically, these are needs-based individualized tests that aim at a holistic assessment of the person and his or her behavior, as well as his or her support needs and requirements for the daily and working routine. In Germany, there is no consensus on which procedure should be preferred.

Table 1. Short selection of tools to assess the abilities of individuals with cognitive impairment
Full size table

In addition to cognitive impairment, most people have other disorders. For example, a combination with physical impairments is common. Collectively, people with cognitive impairment can be classified into different levels of intelligence impairment. The World Health Organization (WHO) defines intelligence impairment as a developmental manifestation, arrested or incomplete development of mental abilities with a particular impairment of skills that contribute to the level of intelligence, such as cognition, language, motor and social skills [9]. These levels describe the intelligence age of a person compared to the age of a child. This shows that a comparison is often made between people with impairments and children.

In Germany, the latest figures from 2016 show that to date 680 workshops belonged to the Federal Association of Workshops for People with Impairments, in which a total of 308.691 people with impairments worked, of whom the clear majority 76% were affected by cognitive impairments (see Fig. 2) [10]. In order to be able to properly assess the abilities of cognitively impaired test users, a longer period of time, expert knowledge and a certain connection with the test users are furthermore needed. This is difficult to achieve as an external person, which is why cooperation with care institutions such as workshops for impaired people is essential. The workshops regularly carry out analyses and tests. As already mentioned, which procedures are used differs from institution to institution, at least here in Germany.

Fig. 2.
figure 2

People with impairments in workshops for people with impairments 2016 by types of impairment [10]

Full size image

2.2 Analysis of the Procedure and Aspects of Usability Testing

Usability testing is a method in which experiments are conducted with users of an application using specific test tasks in order to identify usability problems. Representative users of the application are given specific and typical tasks to perform with the application. The users are observed while performing the tasks. The aim of usability tests is to obtain feedback from future users at an early stage of product development in order to increase usability and thus customer acceptance through product improvements. The procedure of usability testing can be divided into three main steps - preparation, realization and evaluation. The actions inherent in each of the three steps is shown in Fig. 3.

Fig. 3.
figure 3

Own depiction of usability testing procedure based on [11]

Full size image

Next, related work in the context of usability testing with cognitively impaired persons is discussed. In relation to this specific test user group, different guidelines and studies consider various test recommendations with regard to user requirements and needs. A paper by Imke Niediek [12] deals with questions and supporting techniques in interviews with cognitively impaired people. In addition to the aspect of the question format, Tassé et al. refer in their Guidelines for Interviewing People with Impairments [13] to the relevance of further aspects such as processing time and respondent rapport when conducting interviews and represent a certain interviewer etiquette. Even though the question format is an important part of testing, the usability factor, which means the actual testing of applications or interfaces with cognitively impaired people, is missing in the sources mentioned so far. Mostly, such usability tests only involve elderly people and/or people with learning impairments. An example of this is a usability test of information technology applications with learners with special educational needs conducted by Williams and Nicholas [14]. In the same year Williams published Developing methods to evaluate web usability with people with learning difficulties and also provided insights there into “A number of key themes emerged, including the nature of the tasks encountered, engagement, relevance to needs and the role of supporters” [15]. Another test by Williams and Shekhar [16] focuses on testing the usability of a Touch-Screen Interface of Smartphones with people with learning impairments. In view of the current COVID-19 situation, remote tests have regained relevance and can certainly be conducted with cognitively impaired people, as the paper will show in the further course. Conducting remote usability tests with impaired people is by no means new. Already in 2006, Petrie, Hamilton, King and Pavan published an article on this topic, which provides “[…] a set of principles for local and remote evaluations with impaired users […]” [17].

These studies and guidelines show that the inclusion of cognitively impaired people in interviews and usability tests is comparatively rare, but by no means a novelty. The creation and conduction of suitable test concepts brings with it some challenges, but with the right knowledge about the abilities of the user group of cognitively impaired people and the corresponding adaptations to them, it is just as possible as with any other user group. Nevertheless, they are more likely to be involved in interviews, some of which are used to assess their own abilities or to get their opinions on issues. However, the involvement of cognitively impaired people in usability tests is rare and tends to focus on people with learning impairments. Therefore, usability tests with cognitively impaired people who also have severe intelligence impairments seem to represent a research gap. In light of the desired inclusion of people with impairments and the advancement of accessibility, however, this represents an important element in achieving it.

Table 2 lists guidelines and studies on the respective phases of usability testing from which recommendations can be derived. It is shown which abilities of cognitively impaired people can be better taken into account by modifying the procedure of usability testing.

Table 2. Extracted recommendations to the consideration of usability testing with cognitively impaired persons
Full size table

Based on the information obtained from the Related Works mentioned above, as well as through close cooperation and consultation with caregivers of cognitively impaired test users, test concepts for usability tests with cognitively impaired persons could be developed and conducted. Detailed explanations on this follow in the next sections.

3 Method

In the next step, the paper examines the conduction of the modifications for usability tests with cognitively impaired test users within the framework of its own case study, which were determined on the basis of the analysis of the guidelines and assessment procedures for cognitively impaired people. The case study compares the application in an on site test and a remote test and critically examines the advantages and disadvantages.

3.1 Case Study – Modified Usability Testing

Two usability tests were conducted with cognitively impaired users. The on site test was conducted with 6 test users, referred to below as test group A, and the remote test was conducted with 34 test users, referred to below as test group B.

As already mentioned there are no uniform standards to assess the abilities of cognitively impaired people. Regarding the test groups the assessment was based on competence analysis and the Werdenfels Test Battery (WBT) for measuring cognitive-intellectual abilities of people with impairments (see Fig. 4) [5].

“The WTB includes the following subtests: Temporal Orientation, Spatial Orientation, Auditory Memory, Visual Memory, Episodic Memory, Number Knowledge, Series Formation, Quantity Comprehension, Arithmetic Tasks, Factual Tasks, Articulation Language, Nominative Function Language, Vocabulary, Concept Formation, Situation/Content Comprehension, Task Comprehension, Reading/Comprehension, Visual Differentiation, Consistency and Reflection, Fine Motor Skills” [5].

Fig. 4.
figure 4

Extract from the WTB, conducted with one of the test groups (Color figure online)

Full size image

The test users were selected in close consultation with the caregivers. Based on the cognitive impairments of the test users identified by them, a cross-section of possible cognitive impairments within test groups could be presented. However, the caregivers were not only involved in the selection of the test subjects, but also in the entire test conception and conduction. Only in this way was it possible to modify the tests to suit the test groups. As introductory examples, only two modifications are briefly mentioned here:

The phrasing of the test instructions had to be clear, short, in simple words and unambiguous. In order to achieve this, it was necessary to discuss in advance with the caregivers how Easy Language needs to be applied and what vocabulary needs to be used in the institution.

The phrasing of the test questions also had to be clear, short, in simple words and unambiguous. It had to be clarified in advance with the caregivers what evaluations the cognitively impaired test users are capable of. In this case, it was not possible to go beyond a three-scaled evaluation - good/neutral/bad or yes/maybe/no. In the case that test persons could not speak, symbols (tick, cross, thumbs up, thumbs down, thumbs horizontal) in combination with emoticons (smiling, grumpy, neutral) and color gradations (red, green, yellow) were prepared in advance (to open up a communication channel for the test persons. It was also important to ensure that the symbols were familiar and common, as they can differ from one institution to another.

These two examples already show very clearly that the modifications with regard to the cognitive impairments of the test users are necessary in order to make the conduction of a usability test possible. The following section therefore deals in more detail with the conduction of the modifications in the on site and remote tests carried out as part of the case study.

3.2 Case Study – On Site vs. Remote

In the case study, the modifications for usability tests with cognitively impaired test persons were carried out both on site and remotely, and compared to determine the advantages and disadvantages of the two options.

During the on site test, there were always two persons from the test team, one cognitively impaired test user and a caregiver for the test user in one room. During the remote test, one cognitively impaired test user and a caregiver were on site in a workshop for impaired people. Two persons from the test team were remotely connected. In the scope of both tests the app to be tested was made available to the test user on a tablet. The setup of both tests can be seen in Fig. 5 and Fig. 6.

Fig. 5.
figure 5

Test group A – On site test setup

Full size image
Fig. 6.
figure 6

Test group B – Remote test setup

Full size image

In accordance with the preparation of the usability testing procedure mentioned under related works, Table 3 lists the modifications that were made to the individual phases of the usability tests on the part of the on site test and on the part of the remote test.

Table 3. Modifications of usability testing procedure - On site vs. Remote
Full size table

4 Results

After completion of the tests, it becomes clear that usability tests with cognitively impaired people are entirely feasible. There were no drop-outs during the tests, to which the modifications and the involvement of caregivers contributed decisively. The modifications made regarding the common usability testing procedure have proven to be purposeful, but the qualitative evaluation makes it clear that they still need to be fine-tuned to the needs of the cognitively impaired people.

4.1 Case Study – Modified Usability Testing – Advantages and Disadvantages

Based on the previous explanations of the modifications made to the usability tests, one factor has emerged as crucial - the cooperation with the caregivers. This plays an important role throughout the entire usability test procedure, as only they know the full extent of the test persons’ characteristics and abilities. This means that their assessments and support are not only necessary during the preparations, but also during the conduction and evaluation of the tests. Through consultation with the caregivers, the test leaders obtain a comprehensive picture of which cognitive impairments the test persons are subject to and which measures must be taken to compensate them within the scope of the usability test.

With regard to the test preparations, most of the modifications proved to be successful and sufficient. In particular, the formulation of the test instructions and questions in Easy Language with familiar vocabulary and simplified rating options, as well as the reading aloud by the test leader and the decision to conduct the tests in an environment familiar to the test persons, enabled and facilitated the attention and understanding of, as well as the communication with, the test persons. The cognitively impaired test persons were mostly able to understand and implement the test tasks directly at the first attempt. In particular, the use of simplified and familiar terms to make partial aspects of the prototype comprehensible proved helpful (e.g. avoiding Anglicism). However, one aspect that can be better adjusted to the target group is the procedure for obtaining the consent of the test persons to participate in the tests. Although it is correct and obligatory to obtain consent through the legal guardian, it would be motivating for test persons with less severe limitations and, moreover, beneficial for the feeling of independence if they were also given a consent form and had it signed, even if the test persons can only sign with an X. It should be noted here that it must be discussed with the caregiver beforehand which of the cognitively impaired test persons would find this procedure motivating and which would find it rather unsettling.

With regard to the test conduction, it becomes clear that the test leader plays a much more important role than usual in the context of usability tests with cognitively impaired people. For the test leader, a deeper understanding of the test conduction is required, since every aspect, no matter how small, must be taken into account and a multitude of prepared alternatives must be worked with in order to enable interactions and evaluations despite limited communication and combination ability on the part of the test persons. This also includes the decision to allow the presence of caregivers during the usability tests or even make them proxy test leaders. This circumstance encouraged and motivated the cognitively impaired test persons to approach and perform the test tasks without fear. In addition, it gave them enough confidence/safety to openly exchange thoughts with the unfamiliar test leaders. This modification has proven to be a double-edged sword. On the one hand, it simplifies communication and increases the motivation and concentration of the cognitively impaired test persons. On the other hand, despite detailed instructions from the test leader, there is a risk that the caregiver/proxy test leader becomes too involved and unintentionally controls the interaction and assessment on the part of the cognitively impaired test person out of a habit to help him or her. In order to avoid admonishing exchanges between test leader and caregiver/proxy test leader during the test or between the individual tests, gestures or similar should be agreed upon in advance, which test leaders can use to slow down the caregiver/proxy test leader without disturbing the course of the test and the attention of the cognitively impaired test person.

With regard to the technology used in the usability tests, it became apparent that at least the test groups in this case study were very open-minded and interested in any form of technology. This was an advantage with regard to the test object - the prototype - because the cognitively impaired test persons tested it willingly and curiously. However, this curiosity becomes a disadvantage when it is directed towards other technology that is only used to observe the tests. Thus, parts of the technology used can also lead to distraction of the cognitively impaired test persons. The use of technology therefore offers a lot of support in the interaction with cognitively impaired test persons, but must also be precisely adapted to their needs and limitations so as not to appear as a disturbing factor.

4.2 Case Study – On Site vs. Remote – Advantages and Disadvantages

The modifications of usability tests were used in various forms in the on site and remote tests. Based on this, a comparative statement can be made about their advantages and disadvantages.

With regard to the test preparations, the preparations for the remote tests proved to be more extensive and lengthy. Beforehand, ways and means had to be tested to be able to observe the interaction with the tablet on which the prototype of the app was located and to follow the communication between the cognitively impaired test persons and the caregiver/proxy test leader on site. This meant that relevant software had to be installed on the tablets and laptops and the tablets had to be sent to the workshop for impaired people. In order to ensure that the usability tests ran smoothly, the deputy test leader on site had to be instructed in the use of the technology in addition to the test concept.

Even though the preparations for the remote test were more time-consuming, it turned out that the performance of the test persons was much more focussed and consistent than at the on site tests. The remote test offered the advantage that only one cognitively impaired test person and one caregiver/proxy test leader were present in the workshop to guide the test persons through the tests. Since the test persons only held the tablet with the app to be tested in their hands, the laptop via which the scene was remotely connected to the test leader was not noticed at all by the test persons due to a smart placement in the room. In this way, the distraction of the cognitively impaired test persons by the unfamiliar test leaders and by the functioning of the transmission technology, which would have been too complex for the understanding of the cognitively impaired test persons, could be avoided. Thus, at least from the point of view of the cognitively impaired test persons, the test was conducted by only one person, namely a caregiver known to them.

Based on this, the second major difference between the on site test and the remote tests emerges. In the on site test, the test leader was able to guide the cognitively impaired test persons through the usability test themselves, whereas in the remote test, a caregiver from the institution had to act as a proxy test leader (for the reason already mentioned, that the transmission technology in combination with the communication with several strangers would have represented too great a distraction). This gave the on site tests the advantage that the test leader had a direct overview of the scene, could offer interaction alternatives if necessary, and could involve the caregiver sitting in on the test. This was not possible with the remote test. The test leader had to rely completely on the proxy test leader and leave all interaction with the test persons to him. However, even with good preparation of the proxy test leader by the test leader, deviations from the planned procedure occurred from time to time due to inexperience of the proxy test leader (e.g. questions were asked differently or the order was changed). In these moments, the fact that the test leader was not even noticed by the test persons became a disadvantage, as the test leader was not able to draw attention to him/herself and enter into an exchange with the proxy test leader. For the next remote tests, it can be deduced that a communication channel between the proxy test leader and the test leader should be planned via headphones in order to be able to counteract such errors. However, since only one and the same caregiver acted as proxy test leader during the remote tests, the tests became more and more routine. The proxy test leader was a caregiver who was known to all cognitively impaired test persons, even if he was not specifically responsible for them personally. The individual caregivers of the test persons were also remotely connected without the test persons being aware of it. This procedure minimised the effort involved in training a proxy test leader, but nevertheless allowed the assessments of the individual caregivers to be taken into account based on their many years of experience in dealing with the respective cognitively impaired test persons.

In conclusion, it can be said that although the remote tests are more time-consuming to prepare and to conduct, but they cause fewer distractions or uncertainties on the part of the cognitively impaired test persons compared to the on site tests and thus provide a more realistic picture of and assessment on the part of the cognitively impaired persons.

All in all, usability tests with cognitively impaired people are very feasible and also effective. However, compared to usability tests with other user groups, the cognitively impaired test persons could only suggest minimal recommendations or alternatives, as they are not aware of the functional and design potential of the app development. Often, errors in the app are not addressed because they are not perceived as such by the cognitively impaired test persons. They may stumble over the handling, but do not question it. For this reason, the observation of the test subjects and the assessment of their reactions in cooperation with the supervisors is also of great importance for the evaluation of the tests. In this way, the results can be put into perspective. In connection with the closed question formats and the three-point rating scales, which the cognitively impaired test persons were able to take without any problems, it was thus possible to obtain evaluable and interpretable results.

5 Discussion

Due to the diversity of the user group of cognitively impaired people, it is of utmost importance to consider each phase of the usability test in particular and to modify it according to the needs of the user group. Generalizability is only possible to a limited extent, because although the impairments of the test persons can be taken into account in the test design, their emotional stability cannot necessarily be. This means that these challenges can only be overcome with the support of the caregivers - an approach that can only be designed or planned uniformly to a limited extent.

Creating alternatives is an important part of planning and conducting usability tests with cognitively impaired people. On the one hand, these alternatives relate to enabling interaction and communication with the cognitively impaired test persons. On the other hand, they refer to the possibilities offered to them to perform an evaluation and to communicate their opinion. These alternatives can only be tailored to the cognitively impaired test persons in close consultation with the caregivers. In the case of test leader, it can even go beyond mere consultation in that way that the caregivers themselves act as proxy test leaders, as described in the example of remote tests. In this case, the caregiver is briefed in advance by the test leader in order to introduce the cognitively impaired users to the test items and the associated evaluation options as well as to the technology to be used for this purpose.

Contrary to general expectations, the tests (both on site and remote) showed that at least the cognitively impaired test persons in this case study were very open-minded and willing to test the technology provided. It is not perceived as an uncertainty factor but rather as a motivation. Especially in the context of digitalization, the user group of cognitively impaired people should not be neglected. This would make an important contribution to inclusion, which aims at the social participation of all people, regardless of their individual dispositions and starting points [18].

In conclusion, it can be said that usability tests with cognitively impaired people are very feasible if proper modifications are made to the test procedure. Thus, in the sense of user-centered development, usability tests with cognitively impaired people should also become the norm.

Fig. 7.
figure 7

Project and its funding organizations

Full size image