Keywords

1 Introduction

In recent years, interactive computer-based systems have become tools for communication, collaboration and social interaction amongst diverse user population with different abilities, skills, disorders, requirements and preferences in a variety of contexts of use [1]. Indeed, the development and implementation of new systems and methodologies should assure a more user-friendly approach and become a motivational/behavioral solution that aggregates multiple advantages regarding different fields of interest (e.g., rehabilitation of patients) [2]. As such, the needs of the users are becoming increasingly important and digital environments should be accessible and usable by anyone, anytime, anywhere.

Digital inclusion is considered a Human Right, as digital environments provide an unlimited number of services, products and benefits for personal, professional and social contexts. The term “user interfaces for all”, firstly coined by Stephanidis et al. in the late 90’s [3,4,5], represents an effort to overcome known accessibility and usability challenges and should be conceived as a new perspective on HCI. However, “one solution does not fit All”, as there are specific target-populations not able to handle digital environments similarly, mainly due to cognitive or physical setbacks. In this perspective, our top research goal is to provide a user-centered solution for one main audience – people with disabilities – to accomplish specific tasks.

With this goal in mind, we followed a new approach and explored a new area of interest in our research: provide people with hearing loss with a multisensory music experience, through visual and tactile stimuli. Specifically in this study, we present the audio samples recorded, the mobile application developed and the contexts of use.

Indeed, our first step was to analyze previous studies, in which authors use sound as a mean to convey information. Furthermore, after recording some audio samples, we tested them in order to understand which type of emotion could be transmitted to a person with impairments of the auditory system. Hence our focus on haptic interfaces and its capacity to cause vibrations as the mean to provide the user with important feedback. Indeed, the proposal of our preliminary investigation is to create a mobile application, thus providing portability, that allows testing of some audio samples with the multisensory goal in mind. Our ultimate ambition is to be able to convey specific basic emotions, such as fear, adrenaline, anxiety; and even cause more complex feelings like the ones affiliated with the act of dancing or relaxing. Likewise, we developed audio samples to evoke moods of Suspense, Drama and Adventure.

Finally, we pursued a user-centered design methodology and invited a smartphone user with hearing loss to be part of our development process, thus providing valuable feedback regarding the accessibility and usability of the solution presented.

The paper is structured as follow: first, we present a background analysis with a brief statistical context and characterization of hearing loss as a disability. We deepen the concept of inclusive sound experience through vibrations and introduce studies of sound as an inclusive content, i.e., studies that use sound as a mean to convey information in different areas. After presenting a brief overview regarding haptic interfaces and their benefits, we discuss the basis on the theory of emotions, thus underlying our choices for this preliminary study. We then introduce our multisensory experience and describe: the methodology and context of use; the mobile application developed; the audio samples recorded; the results of the assessment and the discussion regarding the initial user evaluation. Finally, we present the main conclusions and point out future work.

2 Theoretical Framework

2.1 Statistical Context and Characterization of Hearing Loss

The World Health Organization (WHO) estimates over 5% of the world’s population has hearing loss (approximately 466 million people). By 2030, there will be nearly 630 million people with hearing loss; and by 2050, the number can rise to over 900 million. Furthermore, nearly 1 out of 3 people over 65 years old and up to 5 out of every 1000 babies are affected by hearing loss [6].

When we talk about hearing disabilities, we must consider four levels of severity: mild, moderate, severe or profound; that can affect one ear or both, leading to difficulties in “hearing conversational speech or loud sounds” [7]. Furthermore, there are four established types of hearing disabilities: conductive hearing loss, sensorineural hearing loss, mixed hearing loss (contains elements of both conductive and sensorineural hearing loss) and central auditory loss [7].

Another relevant aspect worth mentioning are the different expressions used when referring to people with limits in the auditory system. A person is considered to have “hearing loss” if he or she is not able to hear “as well as someone with normal hearing” – hearing thresholds of 25 dB or higher in both ears. Specifically, the diagnosis of people with hearing loss is based on [6]:

  • Adults (15 years and older) hearing more than 40 decibels (dB) in the better hearing ear;

  • Children (0 – 14 years of age) hearing more than 30 dB in the better hearing ear.

Another expression commonly used is “hard of hearing” and it refers to people with hearing loss ranging from mild to severe. This group “usually communicates through spoken language and can benefit from hearing aids, cochlear implants, and other assistive devices, as well as captioning” [7].

On the other hand, “deaf people” is an expression regularly used in the literature and refers to people who have profound hearing loss, i.e., very little or no hearing. In these cases, sign language is used to communicate [7].

WHO describes three main areas of impact for people with hearing loss [7]. First, the functional impact: the individual’s ability to communicate with others is compromised, as the “spoken language development is often delayed in children with unaddressed hearing loss”. This affects, clearly, the students’ learning experience. Second, the social and emotional impact: people with hearing disabilities can feel, or even be, excluded from the most basic communications, leading to feelings of alienation, frustration and social isolation, since it is through the process of communication that we relate to each other, developing our identity. This situation is particularly common among older people with hearing loss. Third, the economic impact: “WHO estimates that unaddressed hearing loss poses an annual global cost of US$ 750 billion. This includes health sector costs (excluding the cost of hearing devices), costs of educational support, loss of productivity, and societal costs” [7].

Regarding communication, not all have the same skills: some individuals communicate using oral language and/or writing and lip-reading, others use sign language, and some are bilingual that use both forms of communication [8,9,10]. The development of communication skills involves various factors, such as: the family profile, which determines the way a child is raised and has contact with sign language; the social context, i.e., the education inclusion policies specific to each country; and the type of deafness and the psychological affectations that may result from the disability [7, 8].

Lastly, WHO has asserted the importance of sign language as well, stating that “family members, medical professionals, teachers and employers should be encouraged to learn signs/sign language in order to facilitate communication with deaf people” [6].

2.2 What is Sound?

Sound is defined as air vibrations that the ear can pick up on and convert into electrical signals, which are then interpreted by the brain. The hearing sense is not the only one able to provide this experience: touch can provide a similar experience. With low-frequency vibrations, the ear becomes ineffective and the remaining sensory areas of the body begin to take on more control over the audio capture. For some reason, we tend to make a distinction between hearing a sound and feeling a vibration, but, in fact, they may be the same. In this context, Holmes affirmed “deafness could not mean that you cannot hear, only that there is something wrong with your biological auditory system. Even someone who is deaf can still hear and/or feel sounds” [11].

Namely, sound is a mechanical wave that propagates longitudinally in physical materials. The speed of sound varies with the density of the material in which it propagates, so the denser the medium, the faster the sound. Indeed, sound waves are classified as: sound – mechanical waves produced by a source that emits human audible frequencies, ranging from 20 Hz to 20 000 Hz; infrasound – mechanical waves where the frequency is less than 20 Hz and cannot be heard by humans (notwithstanding the fact that there are some animals that make and hear sounds at these frequencies); and ultrasound – mechanical waves that have a frequency higher than 20,000 Hz, which means they cannot be heard by humans, either.

As Friedner and Helmreich stated in their study, “the frequency spectrum where hearing and deaf scholars have recently been meeting in order to unsettle the earcentrism of sound studies and the visually centered epistemology of much Deaf studies” is infrasound or vibration lower than 20 Hz [9].

Goodman has even gone further and proposed the notion of “unsound”, referring to the infrasonic and the ultrasonic as zones at “the fuzzy periphery of auditory perception, where sound is inaudible but still produces neuro effects or physiological resonances” [12].

Therefore, these authors motivate the use of “sound as a vibration of a certain frequency in a physical material rather than centering vibrations in a hearing ear. Sound plays, thus, a role in experiences where people with hearing impairment can benefit” [12].

2.3 A Brief Note on Haptic Interfaces

Haptic technology refers to the sense of touch, taking advantage of vibrations and / or forces being applied to the user’s body. This designation contemplates data acquisition and object manipulation by the means of the user’s touch, considered as manual interaction with environments that can be real or virtual [13]. Indeed, studies show that tangible user interfaces may even influence the speed and accuracy of specific age groups when completing basic tasks, as content insertion or manipulation, whether they are children or older adults [14, 15].

These sensorial interfaces should adjust to other interaction paradigms and allow a more intuitive use of the systems multimodally, considering several input / output mechanisms in order to create synergies amongst them [16, 17].

Haptic computing is a field of rapid progress and development. There is a multiplicity of disciplines it can embrace, such as biomechanics, neurosciences, mathematics, software engineering, rehabilitation, product design, among others [13].

This technology is, thus, studied within the scope of people with disabilities, as visual or hearing impairments [18, 19], since the touch is more intimately related to the users’ emotions than any other natural interaction paradigm [20].

2.4 The Theory of Emotions

It is relevant to state that emotions and feelings are often used interchangeably, but they do not refer to the same thing. Indeed, emotions come first, then feelings follow with our bodies’ release of specific chemicals, in response to our interpretation of a specific trigger. Then moods grow from a combination of feelings. Indeed, there are different types of emotions that can influence how we live and interact with others [21].

Throughout the years, researchers have tried to identify the different types of emotions that people can experience. Distinct theories have emerged to help categorize and explain the emotions that people feel.

During the late 1970s, Eckman identified the six basic emotions universally experienced in all human cultures: happiness, sadness, disgust, fear, surprise, and anger [22]. He named them the Big-Six emotions and theorized that not all expressions are the result of culture. Instead, they express universal emotions and are therefore biological. Later, he extended this list of basic emotions to include such things as pride, relief, shame, guilt, embarrassment and excitement.

On the other hand, psychologist Robert Plutchik introduced the “wheel of emotions”. Much like the color wheel, he grouped emotions into common areas illustrated with colors, and defended emotions can be combined to form different feelings, much like colors can be mixed to create other shades [23, 24]. He proposed eight primary bipolar emotions: joy versus sadness; anger versus fear; trust versus disgust; and surprise versus anticipation.

Since then, other theories have emerged, focusing on what emotions make up the core of the human experience. A more recent study suggests that there are at least 27 distinct emotions, all of which are highly interconnected [25]. Thus, rather than being entirely distinct, people experience these emotions along a gradient: complex, sometimes mixed emotions, are a merge of basic ones (e.g., basic emotions like joy or trust can be combined to create love).

2.5 Technological Breakthroughs for People with Hearing Loss

The technological breakthroughs for people with hearing loss have been prominent and the use of technology to provide musical experiences is a recurrent practice for people with central auditory loss, sensorineural or mixed hearing loss [9].

The following studies present technological solutions for this specific target by promoting visual or tactile stimuli in order to provide a musical experience in different forms, whether through sound, vibrations or visual displays. In point of fact, several researches defend the use of vibrations and tactile stimuli, as it can allow the transmission of support information in daily lives activities for people with hearing loss.

Ohtsuka et al. presented a body-braille tool as an information transmission support tool for the deaf-blind using vibrations. They used a vibration speaker to improve readability and obtained a correct answer rate of 85%, even with participants with no experience with a two-point Body-Braille system [26].

Manaf and Sulaiman created a mobile application for scenarios involving fire, which integrated vibration sensing and non-speech visualization to notify hearing impaired students in a controlled situation. Specifically in this study, it was proven that integration of vibration detection increased the level of alertness of the hearing impaired during a fire notification occurrence, and that signals can be an effective tip-off with the visualization of alert images [10].

Yao et al. presented a pair of shoes designed to allow vibrotactile sensing and fulfil the dancing entertainment demand of the hearing-impaired dancers [27]. Overall, it seems that the cerebral response to vibrations reaches a speed and a response identical to the capture of sounds or images. This evolutionary mechanism allows people with deprivation of a sense to adapt and compensate using other senses. Some authors report that deaf people have the sensation of vibration in the part of the brain that other people use to hear [27]. These findings suggest that deaf people receiving vibrations have similar emotions to other people when they listen to music.

The study of Mazzoni and Bryan-Kinns (DATA) explored a glove as a “portable, hands-free, wearable haptic device that maps the emotions evoked by the music in a movie into vibrations.” In this study, authors did not test the solution with people with hearing loss. Overall, results indicated “people are able to associate emotional states to vibrotactile stimuli played at different frequencies and intensities”. Specifically, “combination of low intensity and low frequency would induce in participants a low sense of arousal and a low sense of valence, whereas vibrations at high intensity combined with high frequency communicated to people a high sense of valence and a high sense of arousal” [28].

Petry et al. presented MuSS-Bits (Music Sensory Substitution Bits), an ad-hoc wearable solution that enables deaf people to explore sound from various audio sources (instruments, digital devices or environmental sounds) and receive real-time feedback [29]. The authors also presented a literature review of existing music sensory substitution systems and affirmed that the HCI community explored assistive technology using visual [30,31,32] and vibrotactile [33,34,35] sensory substitution systems to bridge the feedback loop gap for musical activities.

Tranchant et al. tested seven individuals with hearing loss and compared performances of 14 individuals with no hearing impairments in order to investigate beat synchronization to vibrotactile electronic dance music in hearing and deaf people. In the experiment, the first group used a vibration stimulus and the second one the auditory stimulus (no vibration). Results showed there was no difference in performance between the two groups and most participants were able to precisely time the bounces to the vibrations. On the other hand, the hearing group showed a higher performance regarding the auditory condition when compared to the vibrotactile condition. Also, they observed that accurate tactile-motor synchronization in a dance-like context occurs regardless of auditory experience, though auditory-motor synchronization is of superior quality [36].

In the research by Trivedi et al., an affordable wearable haptic device for people with hearing disabilities to experience music was developed. The prototype consisted of Vibrotactile sleeves with bone conduction speakers, providing sensory input of vibrations via the bone conduction speakers. The development process was based on subjects’ surveys and feedback on different assistive technologies. For assessment, the authors developed a visualization system that gives visual clues that represent the given musical notes. User testing results showed that this system can be used to provide a musical experience to people with hearing impairments [37].

Furthermore, the visual stimulus is another sense effectively promoted in the development of adapted or universal technological solutions for this specific target. In previous studies, different solutions were described to provide inclusive and autonomous interaction. For example:

Sridhar et al. presented the relationship between pixel colors and sound type, or illumination pattern and sound type, exists for one-pixel-displays among deaf and hearing persons. Results suggested patterns might be more intuitive when compared to pixel colors; and the position and size of the one-pixel-display seems to depend on the personal preferences and should, therefore, be customizable. They also pointed a preference to two of Harrison’s patterns identified in the study: the Staircase Blink pattern for alarm sounds and the Blink Slow pattern for notification sounds [38].

MyCarMobile [39] is a travel assistance android mobile application for deaf people, and was a solution presented to manage the deaf people’s serious communication problems, where the use of smartphones has been explored as a solution to break communication barriers and enhance their communication, providing access to basic services. In this APP, authors explored the usage of iconographic interfaces in smartphones as a solution for providing further autonomy to deaf people, by applying a model for asynchronous and non-verbal communication through iconography. This solution allowed travel assistance services without involving audio, using an iconographic interface to report road accidents. The authors used a user-centered design approach on the development of the prototype and performed usability tests with eleven deaf users, in order to validate the mobile application. They stated good performance and satisfaction levels of the users that interacted with the application.

3 Multisensory Music Experience

Music is an important part of our daily life. We listen to the radio, enjoy concerts or make music. This high exposure to music makes even children experts in music-listening [11]. In musicmaking activities, this expertise enables humans to compare the created with the intended sound and completes the feedback-loop for music-making (play, listen, evaluate, adjust). However, this is a challenging task for a deaf individual (deaf, deafened or hard of hearing) interested in learning to play an instrument [29]. While those systems are well studied and deliver accurate musical information in real-time, often the input possibilities (sound sources) and portability are limited or pre-defined.

Wearable electronics, such as smartwatches, mobile phones and MUVIB [10] could provide pervasive access to sound through vibrations [29], thus demonstrating that music can be capable of conveying emotions [28].

In this framework, and as previously explained, it is thought that people with hearing loss can feel music through vibrations, as they are processed on the same side of the brain where auditory people can hear. Therefore, with the application presented, we intend to produce vibrations through samples so that users can feel emotions transmitted by music, just like with people without hearing problems.

3.1 Methodology

Following a user-centered design methodology, we prepared eight audio samples through vibrations (low frequencies). These were meant to test if we could convey specific basic emotions, such as: fear, adrenaline, anxiety; as well as incite more complex feelings related to suspense, drama or adventure. Likewise, we developed audio samples to evoke moods associated with the act of dancing or simply relaxing. We decided on these feelings / moods following the assumptions of previous researchers that introduced the complexity of emotions, as explained in the previous section.

For this preliminary study, each audio sample was created with virtual instruments and audio manipulation, and developed with a fundamental frequency, ranging from 20 Hz to 250 Hz. Likewise, we took under consideration the audio speed and repetition, depending on the feeling we wished to stimulate. The sound samples were digitally created, with virtual instruments and audio manipulation, using LOGIC X [40] and Ableton Live [41].

Specifically, the fundamental frequencies used for the different audio samples were: 64–67 Hz, for fear; 74 Hz, for drama; 50 Hz, for anxiety; 108 Hz, for the relaxing mood; 90 Hz, for the dancing mood; 55 Hz, for adrenaline; 61Hz, for suspense; and 39 Hz, for adventure.

Regarding the sound experience, and to achieve a fully inclusive sound experience, we considered two experimental test scenarios, each with different devices for the haptic response. Firstly, the user must be in a controlled room and have speakers that can reproduce the full frequency range. They can use a portable computer or a mobile device. For the first, a wood table or other physical material capable of resonating frequencies must be available. Hands are used to feel vibrations or (for an optimized experience) a sub-woofer, where users can put their feet against in order to feel the vibrations. The second scenario takes advantage of a mobile application (and smartphone) with an audio system and Bluetooth, like JBL or BOOSE, to achieve an autonomous and portable sound experience closer to the daily reality. Through this software, the user’s request is sent to the speakers, or other audio system chosen, and the chosen audio sample is played.

Next, we present the application development and test the audio samples.

3.2 Interface Design

The application was designed with the aim of transmitting emotions to users with hearing loss, through vibrations. Specifically, it was created to be ease-of-use and intuitive for all users, thus following a user-centered design methodology.

Fig. 1.
figure 1

Application’s main screen

Full size image
Fig. 2.
figure 2

Application’s main menu

Full size image
Fig. 3.
figure 3

Application’s sound menu

Full size image

To handle the application, users should read the instructions to understand its purpose and options; and then, start navigating.

After loading the main screen (Fig. 1), the main menu appears, presenting three options: Instructions, Who are we?, Start the experience (Fig. 2). In the “Instructions” section, the application provided information regarding the experience (materials used and scenarios) and the different sound samples. The procedures of the experiment were also described (e.g., how and when to put the hands closer to the haptic device in order to feel the vibrations). Regarding the “Who are we?” section, the bedrock of our study was explained. On the other hand, in the “Start the Experience” section, users could choose from eight sound samples, each representing an emotion / mood. Figure 3 shows the first four sound samples’ menu. After selecting the intended sound sample, the user could sense the corresponding vibration.

3.3 Preliminary User Evaluation

We performed a preliminary assessment session with one user in order to understand their first impression and feedback, and also discover sample problems, if any. Our goal was always to improve the sound experience for people with hearing loss. Next, we describe the assessment carried out in our early study.

Indeed, with the aim of developing an interface capable of helping deaf people, we understand that its validation in a real context of use was important. We did not intend to carry out a profound study on the interface and the overall multisensory experience during this stage, but did acknowledge the value of having a target user participating in the assessment of the interface and its implementation. With a participatory design approach in mind, we invited a user with moderate hearing loss to take part in the entire development process and, thus, better understand and meet our target’s needs. The user was 38 years old, with experience using mobile systems and tactile/haptic interaction on a daily basis, due to their use of a smartphone with those capabilities every day for the last ten years. Naturally, the participant gave their signed consent.

The user testing was performed in a controlled environment, but not in a completely isolated room. Both the mobile device (Fig. 4) with the application and the computer (Fig. 5) were provided to the participant, and they were asked to randomly navigate through the application, encouraging an autonomous interaction. It was, however, mandatory to go through all the options provided. Therefore, the participant needed to test the three menu options described in the previous section (“Who are we?”, “Instructions”, “Start the experiment”). When navigating through the audio samples, each time the user selected a sample and sensed it, they were asked to describe the emotion felt for evaluation purposes, i.e., give feedback about which emotion the audio sample transmitted – ultimately, the goal was to verify if there was any correlation between the emotion selected and the one felt.

Fig. 4.
figure 4

Participant exploring the application in the mobile device

Full size image
Fig. 5.
figure 5

Participant exploring the application in the computer

Full size image

The results were promising. The participant was pleased to enroll in this experience and recognized the importance of the studies on digital accessibility, emphasizing, nonetheless, some aspects that needed improvement.

In detail, some of the samples were not able to correspond to the emotions that the user felt, due to our choice to try and recreate complex emotions, and even moods, in this preliminary stage. Basic emotions, such as anxiety and fear were correctly discovered / felt. However, complex moods, like dancing or relaxing were somehow difficult to explain, and thus difficult to be perceived. Namely, the participant revealed that they could not perceive some samples, as well as not being able to interpret a sample that had three sounds including the fundamental frequency. This situation could be due to the room conditions, as it was not completely isolated.

Another feedback given was that the application was well-designed and very easy to interact with. Also, text and images were easy to understand. The participant could test all the options without difficulties or errors during interaction.

Overall, the participant was excited about the multisensory experience and considered it to be an important step for digital inclusion of people with hearing loss.

4 Conclusions and Future Work

With this introductory study, it was possible to have a glance at understanding how people with hearing loss can feel emotions through vibrations. During this preliminary experience, we set two scenarios for user testing and retrieved feedback from it. Much more work must still be conducted with regards to the fidelity of the samples, but we considered this first approach an important step towards the study of inclusive solutions for people with hearing constraints.

Overall, we verified that some emotions initially determined on the audio samples did not correspond to the emotions reported by the participant. Indeed, this early study shows that simple emotions are easier to translate than complex moods. However, we obtained promising results, as the experience was considered effective and satisfactory.

We are aware of the embryonic nature of our study. Consequently, as future work we feel ready to initiate a methodical strategy for user evaluation with multiple participants, thus validating our approach. We also intend to proceed with the production of other samples and provide a different multisensory experience environment, e.g., project a full immersive sound experience resorting to different sensory stimuli.