1 Introduction

The United Nations (UN) Convention on the Rights of Persons with Disabilities (CRPD) obligates States Parties to ensure personal mobility and independence for persons with disabilities by promoting access to and the development of assistive technology (AT), which typically refers to products and services that enhance daily living and quality of life for persons with disabilities (ATIA 2015). AT for navigation enables persons with disabilities to plan and travel to a destination using a variety of technologies for wayfinding, such as electronic white canes, computerised vision and semi-autonomous robots (Yanco 1998; Levine et al. 1999; Hameed et al. 2006; Faria et al. 2010; Fernandes et al. 2010; Cowan et al. 2012).

Research shows that wayfinding involves decision-making—i.e. planning an action; decision execution—i.e. putting plans into action; and information processing—i.e. sensing and understanding new information for decision-making and execution (Passini 1996; Golledge 1999). Since the 1990s, research on wayfinding has adopted a user-centred design (UCD) perspective, and has argued that wayfinding design involves creating environments where people can interact with complex environments and perceive, select and understand information and environmental characteristics to make decisions and reach destinations (Passini 1996; Carpman and Grant 2003; Gibson 2009; Calori and Vanden-Eynden 2015).

Research has examined the experiences of persons with different disabilities using ICT and AT for indoor navigation and wayfinding (Yanco 1998; Kulyukin et al. 2006; Chang et al. 2008). However, in the last year, ICT developers have made substantial strides in deploying Internet of Things (IoT) devices as part of indoor network navigation solutions (INNS) for persons with visual impairments (Ready 2015; Munkås 2016; Transport for London 2016). IoT refer to interconnected devices embedded in ordinary objects, and INNS use IoT devices to augment the physical environment using audio wayfinding information. These solutions use low energy Bluetooth (BLE) beacons and mobile applications to enable persons with visual impairments to navigate complex indoor environments independently (ITU 2017).

BLE beacons, used in conjunction with mobile applications, broadcast wayfinding information about various elements in the built environment—e.g. entrances and exits, toilets, stairs and other points of interest. Thus, INNS provide a more accurate, precise, efficient and low-cost investment for service providers compared with traditional forms of indoor wayfinding for persons with visual impairments—e.g. personal assistance, braille signs, tactile maps or GPS-enabled mobile phones. Industry analysts have also argued that BLE beacons are in the process of disrupting several major industries (Industry Arc 2015).

As research has yet to examine fully the experiences of persons with visual impairments using INNS, this article aims to fill this gap by investigating the usability of a BLE beacon network that uses audio wayfinding instructions for persons with visual impairments to navigate complex environments. This article asks, ‘To what extent do persons with visual impairments perceive INNS as usable?’ This article uses qualitative and quantitative data from an experimental trial conducted at the Pedestrian Accessibility Movement Environment Laboratory (PAMELA) at University College London. The results show that persons with visual impairments largely consider INNS as usable for wayfinding in transportation stations.

This article proceeds in three sections. First, it details the methods used for data collection and analysis. Second, this article analyses the results in terms of overall usability, effectiveness, efficiency and satisfaction. Third, it discusses the results in relation to further development of INNS, and concludes by summarising the results and suggesting new opportunities for future research and development.

2 Methods

2.1 Sample

This article uses quantitative data gathered from a convenience sample of 36 participants (n = 36). Forty-six persons with visual impairments were initially recruited, but ten participants withdrew from the trial for reasons not stated. Participants identified as blind (n = 32) or partially sighted (n = 4) used different mobility aids including white cane (n = 16), a guide dog (n = 8), both (n = 6), and neither (n = 6). Participants’ ages ranged from 21 to 77. The majority (n = 15) of participants’ ages were between 21 and 29, and seven participants’ were over 65. The majority of participants were identified as male (n = 21) as opposed to female (n = 15) and no participants were identified as non-binary. The majority of the participants (n = 15) had a congenital visual impairment. Other participants acquired a visual impairment from age 2 to 10 (n = 6), age 11 to 19 (n = 11) and over 20 (n = 4). Participants lived with a visual impairment from 1 to 77 years with the majority having lived with a visual impairment from 20 to 29 years (n = 12). Some participants (n = 5) were identified as having multiple disabilities including autism, diabetes, depression, anxiety, epilepsy, hearing loss, emphysema and chronic obstructive pulmonary disease.

Participants also reported their activity, use of assistive technology on mobile devices and their general confidence. While the majority of participants reported that they left the house daily (n = 28), some participants (n = 3) reported only leaving the house four times a week. Participants were also asked about their familiarity with screen reading software on mobile phones. Screen readers use text-to-speech synthesisers to convert text information to sound. Two of the most popular screen reader applications for mobile devices are TalkBack and VoiceOver. The former is available on Google’s Android and the latter is available on Apple’s iOS operating systems. The majority of participants were familiar with VoiceOver (n = 30) and some participants (n = 4) were not familiar with either TalkBack or VoiceOver. Finally, participants were asked about their confidence in completing the trial. The participants reported that they were very confident (n = 16), somewhat confident (n = 14) or neither confident nor unconfident (n = 6).

2.2 Protocol

The trials ran for 2 weeks in the Spring of 2017. Each participant was briefed on the trial’s aims and procedures and informed consent was obtained orally. The trials were conducted at PAMELA where four routes were created to simulate different features of a transportation station. Each route was embedded with a series of BLE beacons and adhered to the international standard for audio navigation (ITU 2017). The BLE beacons were programmed to work with a mobile application to provide audio instructions for wayfinding. When a Bluetooth-enabled mobile device passes near a BLE beacon, the mobile device receives an identity (ID) number. The ID number is sent over the Internet to a database where the instruction is selected that corresponds with the ID number. The instruction is sent back to the device and the screen reader converts the instruction to audio.

Participants were first asked a series of demographic questions and then were taken to the beginning of first of four routes (Fig. 1). Participants were then given a bone-conducting headset and a mobile phone with a demonstration application pre-installed. The demonstration application was developed by Wayfindr for Apple’s iOS operating system, and the source code is available online (Wayfindr 2017). Bone conduction headphones were used because they are more accessible for persons with hearing impairments and they allow the participants to hear both the audio instructions and the ambient simulated sound (Fig. 1, audio simulation).

Fig. 1
figure 1

Route map

Full size image

Figure 1 details the four routes and the environmental features. Route 1 (Fig. 1) begins on a platform and the first beacon instructs the participant to turn right and keep to the right of the pedestrian barrier (Fig. 1, Split). The second beacon instructs the participant to turn right and proceed down the stairs. The third beacon instructs the participant to turn right and that their destination is ahead. Route 2 begins on the same platform as Route 1. The first beacon instructs the participant to turn right and keep to the left of the pedestrian barrier. The second beacon instructs the participant to turn left and the third to turn left and proceed through the wide gate. The wide gate is intended to simulate a wheelchair accessible ticket gate used in transportation stations. The fourth beacon informs the participant that they have reached the seating area on the train. The participant is then instructed to proceed to the information booth by turning left and the fifth beacon informs the participant that they have reached the information booth. Route 3 begins on another platform and the first beacon instructs the participant to proceed up the stairs. The second beacon instructs the participant to turn left and keep to the left of the pedestrian barrier. The third beacon instructs the participant to turn left to reach their destination. The fourth route begins near the information booth. The first beacon instructs the participant to turn right and proceed through the wide gate. The second beacon instructs the participant to turn right and continue down the stairs. The third beacon instructs the participant to turn right to reach their destination.

Approximately six participants completed the trial per day and participants were scheduled to arrive in two groups that met in either the morning or afternoon. Participants used their preferred mobility aid during the trial. Participants were allowed 1 hour to complete the routes with a 1-hour break between routes two and three. All routes were completed in order and the ordering of the routes aimed to minimise the potential for the participants to mentally map their environment and reduce learnability. Eight fixed cameras installed in the lab captured video recordings. A technician who noted their overall performance followed participants. After completing the trial, participants were compensated for their participation.

2.3 Measuring Usability

The International Organization for Standardization (ISO) defines usability in relation to three criteria including effectiveness, efficiency and satisfaction (ISO 2002, 2010). While this article adopts the ISO definition of usability, we recognise that scholars have posed other definitions of usability that include criteria such as learnability (Nielsen 1994). In order to measure usability, this article uses three measures including observation of users, performance-related measures and questionnaires. Observations were conducted on the video recordings to validate the performance measures. The performance measures were conducted by a technician who noted the time it took each participant to complete each route, the number of errors made during wayfinding and the number of times wayfinding was abandoned.

The questionnaires included a series of demographic questions completed before the trial, and after completing each route, the system usability scale (SUS) was administered orally. The SUS is a validated survey instrument for measuring perceived usability of ICT products and services for sample sizes as low as 20 (Sauro 2011). The SUS consists of ten questions scored using a Likert scale from one to five. The results are converted to a score ranging from zero to 100. Scores are typically interpreted based on additional qualitative data. The SUS was supplemented with two additional questions. The first question asked about the participant’s confidence in using INNS for wayfinding in the future and was scored using a Likert scale from one to five. The second question was qualitative and asked about the participant’s overall experience.

3 Analysis

3.1 Effectiveness

Qualitative performance measures were used to assess effectiveness. A technician observed and upon completion of each route documented whether and to what extent the participant navigated the route accurately and completely (ISO 2010). The qualitative data was coded according to themes that emerged from the observers’ comments (Miles and Huberman 1994). Of the 132 records on effectiveness, 67 results showed that the participants experienced a barrier to effectiveness.

The results showed that the barriers to effectively navigating the routes led to a variety of outcomes. Participants sometimes failed to complete the route (i.e. the participant gave up). They sometimes failed to follow (e.g. went to right instead of the left side of pedestrian barrier or turned left instead of right) or complete the instruction (e.g. they did not arrive at the destination or did not follow the instruction). Participants also repeated steps to complete instruction (e.g. participant retraced their steps). They also frequently hesitated during the route—according to one technician ‘the participant hesitated because [they] did not hear any instruction’ and at times experienced confusion or disorientation—according to one technician ‘[the participant] was confused after going [down]stairs’.

The results showed that the timing of the instructions posed the principle barrier to participants’ effectively navigating the route. This article argues that the timing of the instructions refers to when and where, in relation to the desired action, a user receives the instruction. According to ITU (2017), users should receive an instruction 8 ± 1 m before a decision point such as making a turn. Technicians observed that the participants received the instructions sometimes too early and sometimes too late. For example, during one route, technicians observed, ‘[the participant’s] walking was slow and the instructions came early’. Another technician observed ‘the instruction to turn left […] came late and [the participant] had to wait’. Technicians also observed that the participants’ walking speed affected the timing of the instructions. For example, one technician noted, ‘[the participant’s] movement was relatively slow, so the instruction […] was not synchronised’. Another technician noted, ‘instructions were not well synced with [the participant’s] movement’. Technicians also noted that a delay occurred between the participant receiving an instruction and taking an action (e.g. ‘there was time delay between instruction and response’).

3.2 Efficiency

Quantitative performance measures were used to assess efficiency. A technician timed how long each participant completed each route. Figure 2 shows the frequency distribution for the time participants took to complete all routes (Fig. 2a) and each route individually (Fig. 2b).

Fig. 2
figure 2

Histogram of time (in seconds) taken to complete routes

Full size image

Table 1 shows that on average, the participants took 54 s to complete a route. While participants took on average much less time to complete route one (50 s) and route three (47 s), route four took on average the longest (61 s), potentially owing to the distance the participant had to travel. The spread of the scores around the mean of route three (17 s) was concentrated, in comparison to routes two (25 s), one (26 s) and four (32 s). The results do not appear to relate to the number of beacons or instructions given along a route. However, the data, only to a limited extent, provide a useful basis for examining the relationship between the number of instructions and time taken to complete the route. Hypothetically, a higher number of instructions may result in an increased amount of time for decision-making and longer routes. The distance that the participants travel may also explain some of the variation in time taken to complete the route. Nonetheless, the results provide a useful basis for conducting further research on the efficiency of INNS using BLE beacons.

Table 1 Descriptive statistics for time taken to complete routes (in seconds)
Full size table

3.3 Satisfaction

The SUS questionnaire was used to assess satisfaction and it was analysed using boxplots to represent the overall and broken out scores by route (Fig. 3). The quartiles in the boxplot were interpreted using qualitative data drawn from questions about the participants experience using the system. The boxplot shows that the median of all SUS scores for all routes (n = 132) was 85. Research suggests that SUS scores above an average of 68 are considered usable (Sauro 2011). The median SUS scores remained stable across all routes and ranged from 83 to 92.

Fig. 3
figure 3

Boxplot of SUS scores

Full size image

Overall SUS scores in the first interquartile (Q1–MIN) ranged from 75 to 45 (Fig. 3a). Comments for scores in this range typically focused on instruction timing. According to one participant, ‘the instruction should come just before actions are needed’. Another participant noted ‘the instructions weren’t delivered at right place … it’s difficult to know if it’s working or not as no instruction came … feedback of the distance to the next point is needed’. A third participant suggested that ‘the device should be customised [for] faster walking … I didn’t get correct information to the destination’.

SUS scores in the second interquartile (Q3–Q1) ranged from 95 to 75 (Fig. 3a). Comments for scores in this range also noted that the instruction timing posed a barrier to usability. However, participants also commented on their satisfaction with the system. According to one participant ‘apart from the fact [that] I got stuck and the instruction came late, the whole function was good … the instruction that did not arrive confused me’. A second participant commented, ‘it is nice after understanding how to use it’. Another participant also noted their satisfaction stating the system was ‘easy to use and I enjoyed it’ and followed up by noting, ‘it’s better to repeat instructions automatically’. Finally, a fourth participant commented on their satisfaction stating it was a ‘good experience overall’ and stated that the testing area was small and that ‘a larger setting … could be better’.

SUS scores in the third interquartile (MAX–Q3) ranged from 100 to 95 (Fig. 3a). Comments for scores in this quartile were generally very positive—e.g. participants commented ‘good experience’, ‘easy to understand and follow’ and ‘exciting to experience this kind of technology’. One participant commented ‘the instruction[s were] really useful and valuable’ and went on to state that with the system they were able ‘to safely negotiate the stairs’. Another participant noted, ‘the instructions were clear and made me able to get from “A” to “B”’. However, participants also had suggestions for improving the system—e.g. one participant commented, ‘it would be good if [the] instructions sent more detailed information like which side the hand rail is [on]’.

The boxplot also provides evidence of three outlier scores in routes two and four (Fig. 3b). According to one of the participants, ‘The system did not provide information to make a decision; [it] failed three times to tell me where I was’. Another participant stated, ‘It tells me at wrong time and … I couldn’t find [the] stairs as [the] instruction didn’t come properly’.

4 Discussion and Conclusion

The results show that persons with visual impairments broadly perceive INNS as usable. However, in terms of effectiveness, persons with visual impairments experience barriers using INNS. In terms of efficiency, the skewed distribution of the time taken to complete the routes suggests that while most participants took approximately 50 s to complete the routes, several participants took 2–2.5 times as long to complete the routes. The time taken to complete the routes may be related to a person’s specific impairment or other demographic characteristics. However, the data did not show a conclusive relationship. Although the SUS scores revealed a high level of usability, participants’ satisfaction with the INNS was mixed.

Overall, the results suggest that persons with visual impairments experienced barriers using INNS due to the timing of the instructions. This article suggests that future research should continue to investigate the usability of INNS for persons with visual impairments and focus specifically on the timing of the instructions. While this article uses data from a convenience sample of persons with visual impairments, future research could extend the results of this article by investigating the relationship between perceived usability and other variables such as impairment type, age, confidence and experience using ICT. This article also recommends that future developers of BLE beacon hardware and INNS software focus on promoting the reliability and responsivity of instruction timing.