Keywords

1 Context

In 2007, Apple launched its first iPhone with iOs, and in late 2009, the first Android phones were released. Today these are the standards of the personal connectivity market. According to [6] at the end of 2015, the penetration of mobile phones in the world rose to 97%, and the number of mobile devices globally reached 7.9 billion, more than people on our planet. In Europe, 78 out of every 100 inhabitants have a smartphone. Global mobile traffic is predicted to grow by about 8 times between 2015 and 2020. By 2019, mobile video will account for 72% of all global mobile data traffic.

1.1 History. Mobile Devices for Access to Culture

Theater

More than 5% of the population suffers from a sensory disability (auditory or visual). In order to achieve equality in the total integration of people with disabilities [12], it is necessary to provide elements of accessibility to cultural events. For this purpose, techniques such as stenotyping, sign language interpretation or live audio-description have been used [7]. However, the foregoing are too expensive and depend on the availability of an expert at the time of the event. In 2011, the first experience of using smartphones for accessing cultural events, the UC3MTitling system for theatre, was presented [4]. The system allows both captions and the audio description, with two different modules.

The script editor, which uses the performance script in Word format, defines the styles for each of the characters in order to identify them correctly. This identification of the different characters is carried out by means of font, size, color and other text parameters, creating styles that are applied to all the text of that character, allowing subtitles to bring an unlimited number of styles.

Once the script has been generated in the first module, the system creates a file that is used to display the titles of a VGA or HDMI output from a computer or sent over the internet on devices such as tablets or smartphones, as shown in Fig. 1.

Fig. 1.
figure 1

Different devices in which it is possible to view the subtitles transmitted over the internet

Full size image

This was a first step in bringing culture closer to people with disabilities through a smartphone and integrating these devices as an expanded screen of the world.

Films

In September 2013, the WhatsCine system was introduced, which makes cinema accessible, enabling blind people to listen to the audiodescription of the film without interfering with the audio of other viewers. At the same time, it allows the deaf to watch sign language through special glasses or follow the subtitling, all through their smartphone. As it is a multi-language application, captions in any language are available.

WhatsCine operates within a local wireless network that provides coverage to anyone who wants to make use of accessibility. A computer is connected to this wireless network and functions as a server. This server is the main component of the system as it contains the software necessary to manage the audio description. It is also houses the videos with the sign language and the files needed to play the subtitles. Likewise, it manages synchronization with viewers to indicate when the playback is taking place and to show everyone the same content simultaneously, as can be seen in Fig. 2.

Fig. 2.
figure 2

Whatscine system initial architecture

Full size image

Each viewer connects to the accessibility server via their personal mobile device. These terminals receive audio from the audio description, the video with the sign language and the content of the subtitles, all in a synchronized fashion, through the wireless network. The advantage of client terminals is that it is the users’ own intelligent devices (smartphones and tablets) that are used to receive accessibility elements, making implementation of the WhatsCine system even more viable (WhatsCine has been developed for iOS and Android mobile platforms.)

The three main functionalities the system offers are visibly differentiated and separated within the application. They are easily accessible to all people as they are properly labeled so that the accessibility assistants of the terminals themselves can find them. In the case of a blind individual who wishes to enter the audiodescription, they can simply scroll with their finger on the screen of the device so that it tells them what option they are using and confirm with a double click where they wish to enter.

Of course, not all accessibility methods may be required for all events, so if only the audio description and subtitling option are set up on the server for example, the sign language feature will not be visible on the terminals.

Audiodescription

Audiodescription basically consists of providing sound information on all those data, situations and details that are essential for understanding of certain cultural acts and events that only appear visually [13]. It also allows audio description to be received without the need for a special device (such as a frequency modulated receiver), enabling the audio description to be heard from the personal device of the viewer, without interfering with the audio for the rest of the room.

Subtítles (captions)

The purpose of subtitles is to show the dialogues, translated or not, of a scene in a movie or play, along with relevant information that can indicate the sounds produced in the scene. The characteristic feature of the subtitles is that they can be used not only to help hearing impaired people follow a play or film, but also for situations where further information needs to be presented in addition to the visual or when it is necessary to show actors’ conversations or a speaker’s speech in a language other than the one that they speak. Like the audio description, captioning is also set up in the interface of the server software. In this interface, one must specify the files that contain the subtitles along with the name that appears in the client application interface.

An important aspect to keep in mind about subtitles is the format in which they must be stored on the server and properly displayed in client applications. The applications, both in iOS and Android, read these files and are displayed on the device at the pace indicated by the server. The subtitles are contained in XML files according to the Digital Cinema Package (DCP) projection standard [2]. DCP stands for the archive or set of compressed and encrypted files that encompasses the content and associated information of a movie or short film.

The user selects a subtitle language that corresponds to a file stored on the server. The client applications will be in charge of reading this file, interpreting it, asking the server at what time of the session it is and carrying a timer that displays the subtitles read as time goes by. The viewer’s application periodically checks with the server the time of the event and checks that the subtitles are being displayed properly in order to avoid synchronization errors that can be bothersome and cause difficulties in understanding.

Sign Language

Sign language is the last accessibility option offered in the system and is also aimed at those users with hearing disabilities who prefer sign language to subtitles.

In the same way that we proceed with the subtitles, we proceed with the videos of sign language. The video file is accessed from the viewer’s application and the server is asked the time in seconds from the beginning of the session to adjust the playback to the moment it occurs. Playback will then continue uninterruptedly until the user desires.

The only thing that appears on the device screen is the video with the interpreter as shown in Fig. 3.

Fig. 3.
figure 3

Sign language interpreter on a smartphone

Full size image

1.2 Results

Mobile devices provide a solution for the accessibility and integration of people with sensory disabilities into society and culture. Because they are based on personal mobile devices, the need for more complex technical alternatives is avoided, taking advantage of the widespread use of smartphones and tablets.

2 Accessibility in Television via Mobile Devices

Based on the systems explained above, an audio detection algorithm is incorporated into the applications to synchronize with audiovisual material, incorporating accessibility not only for cinema, but also for television. One of the problems with VoD (Video On Demand) platforms is precisely the inclusion of accessibility elements. While in many countries, accessibility elements are not mandatory by law, it should be a moral obligation. In the USA, the Americans with Disabilities Act [9] requires new audiovisual content published on the Internet to have closed captioning.

The inclusion of closed captions only covers part of the population with sensory disabilities, leaving the blind out completely because of not including audio description. It also fails to meet the needs of a significant part of the deaf population, who need sign language for optimal communication.

Given the possibilities of WhatsCine to make audiovisual content accessible with audio synchronization technology, a new app is being developed that enables synchronization with a VoD television, in this case Movistar+ [3]. Since August 2015, the Movistar + platform has been offering film in an accessible format through audio synchronism technology without any additional cost.

The hearing-impaired can choose between Spanish Sign Language (LSE) and subtitles, which offers, in addition to the transcription of dialogues, identification of characters by means of colors, information on sounds and the intonation of dialogues. Neither the LSE interpreter and subtitles invade the TV screen but rather are displayed on the smartphone screen or the user’s tablet.

Similarly, the visually-impaired can use the audio-description system, which works through headphones connected to the smartphone or tablet.

The system designed for television accessibility consists of an app that contains a repository with the accessibility elements of the films or series of the VoD platform. The viewer chooses the accessibility element (adapted subtitles, sign language to audio-description) of the audiovisual material they want to use and the system synchronizes the material from the audio. This synchronization is done on an internal server to which a few seconds of audio captured by the device of the viewer are sent and this server returns the time where the audiovisual material is found.

This system thereby provides a solution for the accessibility and integration of people with sensory disabilities, and as it is based on personal mobile devices, it avoids the need for more complex technical alternatives, taking advantage of the widespread use of smartphones and tablets.

It is true that it may be a nuisance for the spectator who wishes to watch the subtitles or sign language on his or her device and on television, and for this purpose an augmented reality system has been designed so that the hearing impaired can enjoy the accessibility elements more freely. A special lectern has been patented that allows both the subtitles and the sign language to be viewed at the same time as the audiovisual elements in an environment of augmented reality (Figs. 4 and 5).

Fig. 4.
figure 4

Two lecterns simultaneously displaying the two-language subtitles of a television program

Full size image
Fig. 5.
figure 5

Lectern in a cinema showing a sign language interpreter and simultaneous subtitling

Full size image

3 Interconnection with Other Devices

Deafblindness is a combination of low vision and hearing loss, not necessarily complete deafness and complete blindness. Deafblind people have unique challenges in the areas of mobility and communication. It is a unique and diverse disability due to the wide range of sensory abilities, the presence of additional disabilities and the age of onset. A deaf-blind person faces unique difficulties in interacting with the world around him/her.

To do this, a system was designed that from the subtitles that are broadcast in the different channels of TDT in format (DVB-TXT) and with a mobile application will send these subtitles to an app that will communicate with a Braille line.

In Spain, and in other countries, free DTT channels have the legal obligation to broadcast a large number of subtitled programmes. These subtitles can be picked up from the broadcast transport stream and sent to viewers’ devices to be read by voice synthesis or a Braille line so that deaf-blind people can access them.

The system that was developed has the following basic functionalities:

Subtitle extraction module. This module allows the extraction and processing of subtitles (DVB-TXT) collected from the free-to-air broadcast of the DTT channel.

Subtitle Display Management Module. This module will allow the visualization of subtitles extracted by the previous system in the devices of the spectators (ANDROID and iOS).

These developed modules will allow the multiple interaction of users and chains.

The system is currently implemented for all the national and regional channels in the Community of Madrid, which is where the extractor is housed, to which the DTT signal for each multiplex is connected.

Although in principle only the realization of a DVB-TXT subtitle extractor was planned, DVB-SUB subtitles are also extracted, not as text but as an image and an OCR text recognition process is carried out to convert them into text and send them to the central server.

Two GoAll apps have also been developed, one for Android and the other for iOS that allow connection with Braille lines and configuration of the subtitle broadcast.

In Fig. 6, the main menu of the app is shown where you choose the string from which you want to view the subtitles or send to the Braille line (Figs. 7, 8, 9 and 10).

Fig. 6.
figure 6

TV subtitle viewing module input screen

Full size image
Fig. 7.
figure 7

Real-time subtitle display

Full size image
Fig. 8.
figure 8

Accessibility options for connection to the Braille line

Full size image
Fig. 9.
figure 9

Blind person using the system

Full size image
Fig. 10.
figure 10

The system running on the Braille line

Full size image

4 New Possibilities, Mobile Devices as Hearing Aids

Modern hearing aids have improved a lot compared to the technology available only 20 years ago, but they still have a long way to go to serve the approximately 360 million people in the world who have low hearing [5]. Many people with severe hearing loss can understand speech quite well if the sound comes directly to a hearing device eliminating noise and improving some aspects of sound, but today’s hearing instruments have certain limitations such as there is no easy way to connect them wirelessly and reliably to the full range of audio technologies needed today.

Today a smarphone contains all the components needed to build a personal hearing aid. It contains a microphone, a loudspeaker and, in the middle, all the necessary processing power. Right now there are some applications (apps) that amplify sound, but this not only improves hearing abilities, because people who have a hearing deficit also need to modify certain characteristics of sound, and this modification is different for each person.

An app is being designed to allow a customized configuration so that each user can adjust the different frequencies that make up the sound to suit their needs, since hearing loss is usually the loss of sensitivity in some, but not all frequencies, being this the operation of a conventional hearing aid, amplify some frequencies more than others, this allows people with hearing loss and not using a hearing aid can access the sound of television.

This is important because hearing loss is often associated with intolerance to loud sounds and noisy environments, such as a common workroom or a multiple meeting. It may seem strange that a hearing aid should make some sounds quieter, but in this way you can get the hearing-impaired person to hear again.

Hearing aids are associated with improved communication and quality of life, but often unaffordable or inaccessible to many adults. The high cost (~400€ for a hearing aid) and inconvenience (multiple visits needed for installation and adjustments) limit access for many adults.

Hearing loss in older adults is highly prevalent and doubles every decade, beginning at age 40 [8]. Age-related hearing loss begins gradually and may progress for years before an individual begins to experience subjective hearing difficulties [1]. In addition, hearing disorders tend to initially manifest themselves in difficult listening situations, such as working group discussions or conversations in a noisy environment, such as a restaurant. The gradual progression and common occurrence of age-related hearing loss leads many people to communication difficulties [11].

Less than 20% of people with hearing loss use hearing aids, which highlights the need for new approaches to the provision of listening technologies and services to reduce the gap between those with hearing loss and those who are able and willing to access care.

According to the study carried out by [10] “Personal Sound Amplifiers for Adults with Hearing Loss” the main hearing devices recommended for the direct consumer and their main technological and user characteristics, where the market leaders are evaluated: Bean T-Coil, CS-50+, Tweak Focus, Soundhawk and Songbird all have a price above 250 dollars”.

The main innovation of the project being developed: AudiSmart, is to make a hearing impaired person’s hearing aid available on their own smartphone, not limited to a general amplification of the sound, but through specifically designed algorithms, as background noise increases, the gain is gradually reduced to keep production at a manageable level. After all, when the background is noisy, soft, loud sounds are masked, and the amplification only serves to make the noise more annoying. A high gain does not offer benefits in noisy situations. The system will effectively and customize the audio signal to suit the particular needs of each individual.

AudiSmart is not intended to be a simple sound amplifier, it will be a sound amplifier application, the experience of the SoftLab research group in sound processing as well as the experience that has allowed it to develop and evaluate systems such as UC3Mtitling, WhatsCine or GoAll, have allowed it to develop and evaluate a pre/processing algorithm for acoustic stimuli. It is important to think of AudiSmart as the app-based implementation of an algorithm.

5 Conclusions

The use of mobile devices and the internet has enabled us to develop all these technologies as free end user apps without any cost for the disabled individual.

The actions carried out in the SoftLab research group since 2006, when work began to promote accessibility for people with sensory disabilities. Have been marked by the awareness and development of systems that allow access to culture and education for people with sensory disabilities in all aspects. The latest project under development for the collective of deafblind people is noteworthy, as it has always been the maxim of the research group that although the number of people who with a disability is minimal, we must always find a solution so that they can have to culture.

As can be seen, all of the progress made in audiovisual accessibility has been come about by seeking to create viable and useful systems for people with disabilities, such as those presented in this article, all of which are in operation.