Keywords

1 Introduction

Environmental sounds can provide important information about surrounding activity, e.g. on a city street or in the workplace. Sounds are often part of societal conventions for conveying important information, such as drivers honking to alert another driver, alarms that indicate important times and emergencies, and announcements broadcast through loudspeakers at airports [1]. In addition, the activity and presence of other people may be sensed through sound, and many electronic devices (e.g. home appliances) may indicate their status through sound [7]. Recognizing many of these sounds, however, can be challenging for Deaf and Hard-of-Hearing (DHH) individuals. In this study, we expand upon prior work by investigating the perceptions of users of assistive listening devices in regard to the importance of sound awareness in various settings.

Many assistive technologies for DHH individuals have been optimized for speech sounds to support verbal communication, including assistive listening devices such as hearing aids and cochlear implants [7]. However, even for speech related sounds, there is high variability in the effectiveness of those devices, which may be influenced by a number of demographic factors such as the age in which the wearer first obtains the device [2]. In addition, extensive training may be required to obtain a benefit [4]. Furthermore, users of assistive listening devices, e.g. hearing aids, often report challenges in interpreting sound direction, report sensitivity to background noise, or report missing sounds coming from specific directions [2].

In addition to assistive listening devices, specific techniques – such as devices which can trigger flashing lights when the sound of an infant crying or doorbell ringing is detected – can be used to increase awareness of specific sounds. However, prior work has found that these technologies that are specific to one sound are not widely adopted for various reasons, e.g. the monetary cost of purchasing many devices each for one specific sound [7]. Thus, researchers have suggested the use of applications for existing personal devices (e.g. smartphones and smartwatches) that could provide awareness for an array of non-speech sounds [1, 4, 7].

Various prior work has examined the preferences of DHH individuals for different aspects of such applications, including the form factor [3, 7] and the importance of identifying different types of sounds in those environments [1, 3, 7]. Notably, prior work has revealed that DHH users’ perception of sound awareness technologies often varies along some demographic factors, e.g. the individual’s preferred method of communication [3]. Thus, motivated by these prior findings and the limitations of assistive listening devices outlined above, in this work we focus on the preferences of a specific group of DHH users for sound-awareness technology: current users of assistive listening devices. In particular, we investigate what challenges and requirements they report for identifying sounds, which types of sounds they are interested in being aware of in different environments, and what information or properties of these sounds they are interested in knowing, e.g. where it comes from or how loud it is.

The contributions of this work include empirical information about challenges, needs, and requirements in regard to sound awareness technologies for this specific sub-group of DHH individuals. Furthermore, our work extends prior research by identifying user preferences as to the design of these technologies, e.g. which types or aspects of sounds DHH users care about, and whether the setting or form factor influences their preferences.

2 Prior Work

There has been prior work on the preferences of DHH users for various methods of obtaining sound awareness [e.g. 5–8]. Prior work has also explored various form factors for such technology (e.g. smartphones, smartwatches, head-mounted displays) and notification methods (e.g. haptic, visual). Prior researchers have investigated not only which sounds may be of interest [1, 7], but whether the environment or social context influences these preferences [1, 3]. Notably, prior work has revealed that DHH users’ perception of sound awareness technologies may vary along some demographic factors, e.g. users’ preferred method of communication [3].

2.1 Form Factors

Early work by [7] had explored different form factors including stationary (e.g. PC screen) and mobile ones (e.g. PDAs). In this early work, participants indicated preferring smaller form factors, such as a PDA or using only part of a PC screen. However, with the emergence of mobile technologies such as head-mounted displays (HMD), smartphones and smartwatches, more recent research has focused on the use of these technologies [1, 3, 6,7,8]. Jain et al. found enthusiasm for the use of HMD for sound-awareness [6]. However, more recent work by Findlater et al. found that smartphones and smartwatches score higher on social acceptability, usefulness and overall preferability than HMD, while HMD scored higher for captions (for transcribing speech) and easiness to glance [3]. Because in this work we are focused on non-speech sounds, we excluded HMD when asking users to imagine sound-awareness technologies and focused only on smartphones and smart-watches.

2.2 Environments

The early work of [7] had identified different environments in which sound-awareness may be desirable for DHH individuals. Environments that were identified were the home, the workplace, but perhaps most importantly, when someone is mobile (either when walking or when driving). Expanding on the findings of [7], the preferences of DHH individuals have been further investigated in [1], suggesting that DHH users’ interest in different types of sounds varies according to the setting. The importance of these environments in the overall interest of DHH individuals in sound-awareness technologies in general has been further highlighted in [3], who found that this interest is also influenced by what other people are present in each environment. Thus, in this work, we focus on how various environments affect users’ preferences.

2.3 Demographic Factors

In addition to the environment, prior work has also identified how different demographic factors influence the preferences of DHH individuals for sound-awareness technologies. For instance, researchers in [1] had identified that an individual’s identity (i.e. as deaf or as hard-of-hearing) influenced which sounds they would be interested in. Furthermore, researchers in [3] identified that their participants’ preferred mode of communication (i.e. oral, sign language, or both) also influenced their preferences. Thus, in this work, we focus on another key demographic sub-group of DHH users, namely individuals who currently use assistive listening devices.

3 Research Questions

Focusing on people who are DHH and who are current users of assistive listening devices, we investigate:

  1. (1)

    What are the challenges and requirements for identifying sounds, as reported by DHH users of assistive listening devices?

  2. (2)

    In various environments or settings, which types of sounds do DHH users of assistive listening devices report as being most important?

  3. (3)

    What information about sounds are important to DHH users of assistive listening devices and how should that information be conveyed?

4 Methodology

To investigate our research questions, we conducted an online survey with DHH participants, all of whom regularly wore cochlear implants and/or hearing aids. The survey consisted of five sections, containing a mix of closed-ended and open-ended questions. The first section focused on demographic questions about their DHH identity, gender, age, and the number of hours a day they used their assistive listening devices. The second section consisted of open-ended questions focusing on their use of assistive technologies, the challenges they face when identifying sounds and features they would expect from a technology that provides information about sounds. The third section consisted of closed-ended questions focused on how they typically obtain information about the environment and how often they miss sounds in different environments (e.g. home, school). The fourth section contained Likert-type questions asking participants to indicate how much they cared about sound-awareness in different environments and multiple choice questions in which participants selected which sounds they cared about in those environments from a list identified in [1], along with a write-in “other” option. Lastly, a fifth and final section asked participants to indicate which aspects of sounds were of importance to them using a 6-point scale.

4.1 Procedure

The survey was hosted using Qualtrics and took approximately 15 min to complete. Participants in this IRB-approved study were first provided with an informed consent form. After this form, participants were taken through the five sections of the survey. At the end of the survey, participants were offered an opportunity to enter a raffle to receive a $100 gift card as a compensation for participating in the survey.

4.2 Participants and Recruitment

Participants were recruited by word of mouth, social media posts on Facebook community pages, and posters in bulletin boards across the Rochester Institute of Technology, the National Technical Institute for the Deaf, and the surrounding community. A total of 38 participants were recruited. Participants’ ages ranged from 19 to 50 (μ = 24.6). Twenty-three participants identified as deaf/Deaf, while the other 15 identified as Hard-of-Hearing. All participants regularly wore cochlear implants and/or hearing aids. Participants reported that they used these devices between 2 to 24 h per day (μ = 10.5).

4.3 Data Analysis

The results of the open-ended questions were analyzed through a coding system shown in Table 1 and through affinity diagraming. During the coding stage, the first author identified themes in the data, later developing categories based on the resulting themes. Then, the data was organized using an affinity diagram (Fig. 1), sorting it based on the categories generated through the previous step. The results of the closed-ended questions, in turn, were analyzed through descriptive statistics, providing percentages for the scale-based questions and counts for the multiple-choice ones.

Table 1. Coding system used in our analysis.
Full size table
Fig. 1.
figure 1

Affinity Diagram summarizing open-ended responses from the survey

Full size image

5 Results and Discussion

In this section, we present and discuss the results of the survey, and we report the data from all 38 respondents. This section is organized by first discussing the open-ended response data, which is followed by the results from the closed-ended items.

Two of the categories identified from the open-ended data included how well participants could hear sounds in their environment currently, how their assistive technologies improved their hearing of such sounds, and any remaining challenges they faced. These categories are summarized in the affinity diagram shown in Fig. 1.

We noted a wide diversity in the responses for how well users reported being able to hear sounds in their environment, and in the challenges they currently face when recognizing sounds. Some participants reported being able to hear all environmental sounds when using their assistive listening devices. For example, participants discussed their hearing abilities when using assistive listening devices in terms of what percentage of the sounds in the environment or which sounds they can hear:

“Wearing hearing aids, I can hear about 70–85% of sounds in my environment.” (P18)

“I wearing my two hearing aids and can hearing the sound most of the fire truck, police siren car, train, teapot whistle, and a dog barking.” (P37)

In turn, as shown in Fig. 1, the major challenges discussed by participants could be grouped into some comments relating to the location of a sound or relating to the pitch/frequency of a sound. For instance, some participants indicated specific difficulty with high-frequency sounds, and others commented on difficulties they experienced in identifying the location of sounds or in pinpointing individual voices when in a group of people. For instance:

“Moderate difficulty. Sometimes I can identify the noise from a location, but if it’s quieter, I have a hard time to identify.” (P13)

“Always. It’s difficult for me to pinpoint the location a sound is being made.” (P28)

Participants were presented with the idea that future technology may be able to detect sounds in their environment and make them aware of such sounds and their characteristics. Participants were asked about their requirements for such future sound-awareness applications, as shown in the final category included in the affinity diagram shown in Fig. 1. For instance, participants discussed their requirements in regard to the usability of such an application; they also provided recommendations about the type of form-factor for such a device (e.g. smartphone, smartwatch) or the different forms of feedback such an application could incorporate (e.g. haptic or visual). Users also indicated that they were interested in being able to personalize which sounds they would be notified about – this finding was in accordance to the results in [3], which had found that users wanted the ability to “filter” being notified about certain sounds. Some comments from participants illustrating these requirements include:

“Create an app that can detect sounds AND be used via a smartPHONE … not just limited to a smartwatch. Combination of lights, visuals, and vibrations should be used otherwise the user may mistake it for a random text message, email alert, etc.” (P09)

“I would love the app to be ‘easy to use’ and set up - you could have lights for certain sounds (e.g. for alarms) or vibrating options for other preferred sound alerts.” (P31)

“Vibration and Lights are important. So, the app should feature these…” (P21)

The remainder of this section focusses on responses from participants to closed-ended question items on the survey. As shown in Fig. 2, when asked about how they currently obtain information about sounds in their environment, 53% of participants reported using hearing aids “all the time” and 32% reported using cochlear implants “all the time.” Perhaps surprisingly, given that our participants were regular users of assistive listening devices, our participants reported that they still often asked friends, family, or other people to inform them about environmental sounds. Relatively few users indicated that they currently used mobile-phone applications or smartwatch applications to receive notifications about sounds in their environment.

Fig. 2.
figure 2

How often participants used various sound-awareness methods

Full size image

We next asked participants about how often they believed that they missed important sound information, in various settings. The majority of participants reported missing important sounds at least once per day when: mobile and looking at their phone, at work/school, or at home, as shown in Fig. 3. Of course, various factors may affect which types of sounds participants miss most often: For instance, in open-ended comments (summarized in Fig. 1), participants reported having particular difficulty in identifying sounds at certain frequencies (e.g. high or low) or in determining the location of sounds.

Fig. 3.
figure 3

How often participants missed sounds of interest

Full size image

Participants also indicated how important it is to be aware of sounds in the environment in various settings, including: when at home, when at work/school, when mobile (e.g. in a city), or when at a restaurant. Figure 4 indicates that our participants reported greater interest in being aware of sounds at home, at work/school, or when mobile.

Fig. 4.
figure 4

The importance of sounds in different environments

Full size image

For various settings (home, work/school, when driving/walking), we asked participants to indicate which sounds they would be most interested in being aware of (as shown in Fig. 5). Given prior research on DHH users’ interest in smartphones and smartwatches [3], for the driving/walking setting, we also asked participants to rate their interest in being notified about sounds, when using each of those form-factors (smartphone or smartwatch). Having this information about user interest in various form-factors may be useful for future researchers, especially since our participants’ open-ended responses, congruent with prior work [3], revealed that DHH users may want to filter which sounds an app recognizes. However, this filtering may depend upon which form factor is being used.

Fig. 5.
figure 5

Sounds of interest sorted from highest to lowest at a) home, b) work or school, c) when driving or walking with sound-notification via smartwatch, d) when driving or walking with sound-notification via smartphone

Full size image

Finally, we asked participants which aspects of sound were of greatest interest to them (e.g. loudness, location) and how they would want to receive such information (Fig. 6). Since sound-awareness technologies may be able to provide users with information about some of these properties, it is therefore important to identify which are of greatest interest to users. Congruent with the prior findings of [1] and [3], most of our participants wanted to know where sounds come from, how urgent a particular sound is, and how certain the sound-awareness technology was about the sound.

Fig. 6.
figure 6

Importance of different aspects of a sound when the app detects one

Full size image

6 Discussion, Limitations, and Future Work

This paper has presented an online survey among 38 DHH users of assistive listening devices, to understand their interests and requirements in regard to sound-awareness technologies. The findings of our study are largely congruent with those of prior work, e.g. [1, 3]. Our work extends on this prior research by focusing on a specific sub-group of DHH users: individuals who currently use assistive listening devices. Since prior work had found that interest in sound-awareness varies along some demographic characteristics [3], studies of this nature that focus on a particular subset of the DHH population are valuable for confirming whether findings hold true among particular groups.

Our study found that participants reported often missing sounds in their environment, and they reported that they often relied on other people to obtain information about such sounds. The degree to which our participants assigned importance to being aware of specific sounds varied according to the environment and the form factor of the sound-awareness application. Lastly, our participants indicated that both the location of a sound and urgency of that sound were of particular interest to them, and they also indicated that they wanted to know the confidence of the sound-detection application – findings that are congruent with prior work in this area [1, 3].

There were three main limitations of our work. First, we only looked at users of assistive listening devices. Future work may include a comparison of both users assistive listening devices and people who are DHH who choose not to use these devices, which may help determine whether the decision of using assistive listening devices influences their preferences for other sound-awareness technologies. Second, our study may not have been sufficiently powered to determine whether there were any statistically significant differences in our data. Lastly, while we obtained some initial data about preferences for an application that would provide sound-awareness, we had asked users to imagine such an application. In future work, we may obtain more detailed requirements from users using assistive listening devices by showing them prototypes of sound-awareness applications informed by our findings in this work, as well as by prior work in the area.