Keywords

1 Introduction

According to World Health Organization (WHO), approximately 466 million people worldwide suffer from disabling hearing loss and approximately 95% deaf/deaf children are born to hearing impaired parents. WHO estimates that 900 million people will suffer from hearing loss in 2050Footnote 1. According to the Italian Association for Deafness Research (AIRS), in Italy about seven million people suffer from hearing disorders, more or less severe. Most of them use the visual-gestural communication channel, through the Italian Sign Language (LIS), to communicate. The LIS, like any other Sign Language (SL) in general, is a visual language that uses hand shapes, facial expressions, gestures and body language, consisting of a structured and organized set of signs, with grammar, syntax, and its own morphology. It is important to note that deaf people who use SL to communicate, rarely wear advanced cochlear implants. As a result, they often encounter serious difficulties in interacting with the auditory world. Furthermore, a survey conducted by the Gallaudet Research Institute (GRI) showed that most deaf people also have an additional disability; among these the most common are intellectual or learning disabilities, visual impairments and cerebral palsy [1]. As a result of the presence of coexistent or comorbid disorders, the effects of a disability can be aggravated, creating specific and unique needs for the individual who has a further disability in addition to deafness. Communication is one of the most affected areas by the interaction of these needs [2].

In this context, simple daily activities can be difficult for people with severe hearing problems due to the considerable communication effort required. A deaf person or a person with severe hearing loss can face hard challenges in daily communication. For example, they may not be able to hear announcements in public spaces such as train stations, bus stops, airports; attending an interview to get a job may be very difficult; entertainment resources such as public theatres and movies may not be very valuable, not all restaurants, banks, hospitals, shopping malls provide teletypewriters, interpreters or visual warning systems [3]. Hence, simple daily life interactions, such as an administrative or medical interview, may become significantly difficult for a deaf person who is constrained to rely on the use of SL to communicate. This is especially true for deaf people who primarily use SL to communicate. In this case, an SL interpreter is usually needed to support communication between a hearing person and a deaf person. SL interpreters are professionally qualified to translate between national language and SL. Interpreters are booked for professional meetings, interviews, training and conferences. The cost of an interpreter varies depending on the interpreter, the agency and the place where the service is requested. A minimum number of hours and travel expenses must be covered.

In recent years, the need for the Public Administration (PA) to be accessible and inclusive has become more and more a priority. Therefore, having tools and services that can be used to enable deaf citizens to communicate easily is becoming increasingly crucial [4]. For deaf people who only use SL to communicate, an SL interpreter should always be present to support communication with a hearing person. However, having an SL interpreter available in presence at any time is a challenge for a deaf person, for both logistical and cost-effective reasons [5].

For the PA, a live SL interpreting service available at any time would be very useful and inclusive for deaf citizens. However, the presence of an SL interpreter in the PA offices cannot always be guaranteed. As a result, they can only offer limited access to their services since an SL interpreter is missing, unless the deaf citizen is accompanied by a personal interpreter. Furthermore, due to the restrictions on the Covid-19 pandemic, a limited number of persons is allowed in presence in a room. This could lead to additional difficulties for people with severe hearing problems. In this context, new technologies and network communication infrastructures can play an essential role in improving communication between deaf and hearing people.

In this work, we propose a communication tool to support deaf people and people with severe hearing problems in daily activities and in communication services. In particular, we present the LISA (Lingua Italiana dei Segni Accessibile - Accessible Italian Sign Language) prototype, which has been specially designed to be used in a PA context. This tool allows communication between a deaf citizen and a PA operator by establishing a remote SLIS. Thanks to this tool, a remote communication is arranged between three people: the deaf citizen, the PA operator and the SL interpreter. Indeed, the deaf citizen and the operator can meet at the PA office while the SL interpreter can reach them via the Internet, being connected remotely. Usually, a deaf person requires to be accompanied by an SL interpreter to interact and communicate with a PA operator. LISA aims to overcome this aspect, offering a remote SLIS to facilitate the deaf citizen’s interaction with the PA office, guaranteeing real-time interpretation in compliance with Covid-19 restrictions.

2 Related Work

Interaction in a verbal world is a challenge the deaf person faces every day in a predominantly hearing society that often limits their personal contacts and activities. Investigations into ICT solutions for communication between deaf and hearing people are on the rise to explore possible assistive tools [6]. The video communication system E-LISIR (Evolution of the Italian Language Signs with Interpreter on the NetFootnote 2) offers a video call connection service to a video center where SL interpreters can translate in real time from the LIS to the spoken language and vice versa. E-LISIR is a free service that can be activated via an application on a set of preconfigured tablets (located in Rome, at the university and in the main hospital). The number and location of available tablets represent the main limitations of this application. In fact, only deaf users who live in Rome and visit predefined locations can access and benefit from this application.

CGS - Global Communication for the DeafFootnote 3 (Comunicazione Globale per Sordi) is very similar to the E-LISIR app: it offers limited service to deaf people living in Rome. Unlike the E-LISIR app, the user can install the CGS app on their mobile device. However, the CGS app has some critical issues: the user is out of control since there is no navigation menu and some usability problems have been detected by users.

Another video interpreter service with a remote interpreter is Veasyt Live!Footnote 4. This tool offers interpreting service in twenty-five vocal languages including LIS. The interpreter can be contacted via (1) the “Immediate Service”, an interpreter is available in a few minutes; (2) by booking for the next few days. This service can be used via web and mobile apps.

To facilitate communication between hearing and deaf people, there is considerable research in the development of wearable tools and automatic translation systems from verbal language to SL [6]. However, these systems often operate in a very narrow domain (for example, systems designed to assist in the completion of a transaction between a post office (PO) employee and a deaf customer) [7] or wearable technologies often mean that the deaf person should wear numerous or very conspicuous devices (e.g., gloves with sensors, helmet having a camera with infrared filters) [8, 9]. This could potentially pose a stigma to the hearing impaired person.

Our prototype aims to be a tool that can be used freely in any place, accessible from any platform regardless of the operating system of the different devices available to the deaf user, without the need for advanced digital skills. Also, PAs usually don’t use mobile devices to work. Therefore, a cross-platform application can bring benefits to different contexts and users.

3 Use Scenarios

To show the possible use of the LISA application, some scenarios describing possible real situations are reported in the following.

Scenario 1.

Alan is a deaf user who goes to a municipal office to ask some questions. Alan does not use any smartphone or tablet. In this scenario, the municipal employee will have to start the video connection with a remote interpreter. The following steps are required: (1) Log in to the LISA application on the PA staff’s device using their credentials (defined in a previous account registration). (2) Request an SL interpreter via the app. (3) As soon as the interpreter is available, the audiovisual connection will take place.

Scenario 2.

Betty is a deaf user who goes to a municipal office to ask a few questions. She brings her smartphone with (possibly) the pre-installed LISA app. In this scenario, the municipal employee will have to perform the following operations: (1) Provide (if necessary) the user with an information leaflet with instructions to access the application from their device. (2) Wait for the user to initiate the video connection with a remote SL interpreter. Betty, on the other hand, must: (1) log into the app from their smartphone using the credentials created during the account registration; (2) request the availability of an SL interpreter through the application. As soon as an interpreter is available, the audio-video call will take place. Alternatively, Betty could have booked an SL interpreter for this appointment (i.e. for a specific day and time).

Scenario 3.

Charly is a deaf person who goes to the PA offices bringing his tablet with him. Some days before going to the appointment, Charly books an interpreting service for the specific day at the specific time using the LISA tool. The day before going to the office, he wants to ask the booked interpreter a few simple questions in advance. So, the deaf user has already registered an account in the app and has already booked an interpreter for his appointment. Now, Charly wants to contact the booked interpreter via short messages, but, unfortunately, he is not familiar with the written text. So, Charly would like to communicate via SL. Thanks to LISA, he can use the chat service based on simple SL words. Therefore, he can: (1) log into the application; (2) open the appointment section showing the booking with the interpreter; (3) write a message to the interpreter using GIFs that represent words or phrases in SL (see next section for more details).

4 LISA Prototype

LISA is a multiplatform application designed to be used on both a mobile and desktop operating system. The prototype was developed as a Progressive Web App (PWA) in order to have a single application that can be installed on both mobile and desktop platforms while maintaining a single version. This type of solution can be very useful and relevant for PA offices that usually do not use mobile devices, while the citizen or the SL interpreter use their mobile device. This feature makes LISA different from the tools we previously analysed in our study.

The LISA PWA prototype can be used both from the user’s mobile device and from the work device of PA operators. The main functions offered to the deaf user can be summarized as: (1) booking an appointment with an SL interpreter; (2) writing short SL messages to an interpreter; and (3) make a video call with a previously booked SL interpreter or available upon request. A PA operator can (1) call an interpreter when a deaf person arrives at the office without any assistance.

4.1 The Prototype Design

The progressive structure allows you to combine the experience and skills to which users are accustomed to on native web apps, leveraging the strengths of both: the user can install the LISA PWA on the home screen of the device and access the app without using the browser, providing a full-screen interface. The LISA PWA is, therefore, able to function independently of an operating system, browser or device type and automatically adapts to the graphical user interface (GUI) of the device.

A basic style of flat web design was used in designing the GUI: a minimalist template with simple elements, typography and flat colors.

Fig. 1.
figure 1

LISA material design components

Full size image

The layout of LISA is well structured to encourage users to focus only on the content and their goals, avoiding elements of distraction. As shown in Fig. 1, particular attention was paid to the choices of colors, fonts and navigation systems in order to inspire users with confidence, promote readability and complete understanding of the information while remaining visually appealing and accessible [10].

4.2 Booking an Appointment

The LISA PWA allows the deaf citizen to make a remote video connection with an SL interpreter, who translates in real-time SL into verbal language and vice versa, i.e. the conversation between the deaf user and the PA employee. To this end, different roles are defined in the LISA tool.

The LISA prototype offers a multi-role environment: the user can select the appropriate role when accessing the app: “user”, “interpreter” and “operator”.

After logging in, the user is redirected to their Dashboard which provides an overview of the available sections/actions. As shown in Fig. 2, these sections (accessible via shortcuts) allow the user to interact with the system and perform operations.

Fig. 2.
figure 2

Logical diagram of the architecture of the LISA prototype

Full size image

For example, in the “Find an Interpreter” section, the user can get an overview of the interpreters registered in the system. As a result, the user can select the desired SL interpreter, for example an expert in a particular field or topic. For this reason, a filter by areas has been provided which, based on the scope of the selected appointment, retrieves interpreters with the corresponding area of expertise. Areas of expertise include legal, medical, educational, cultural, economics, tourist, technological. Once an interpreter has been selected, the user can read their profile, write a message or book an appointment with that interpreter. In summary, the deaf citizen (user) can book an appointment with the selected and available SL interpreter in order to receive an SL translation for a particular topic (legal, economic, educational, health, cultural, tourism, technology), on a certain day, at a given time via a videoconference meeting.

4.3 Chat Functions

In addition to the booking function, LISA provides users with a chat tool (Fig. 3) to communicate in real time with the interpreter. An innovative feature of this tool is that it allows the deaf user to communicate, in addition to the written mode, using GIFs that represent words and phrases in LIS [11]. This format was chosen because it has the ability to reproduce the visual-gestural characteristics of SL in an infinite and silent loop [12].

The development of this chat tool takes into account the difficulties related to the production of writing by deaf users (with or without additional pathologies). In light of the fact that further methods may be needed that seek to reduce, moderate and compensate for temporary or permanent difficulties of people with severe communication disorders, both on the expressive and receptive front, the use of the Augmentative Alternative Communication (AAC) methodology can overcome communication problems and related barriers and help these users in their social needs in life [13].

In particular, for deaf people, a functional technique of AAC Representational Systems is the Picture Exchange Communication System (PECS), which is a communication strategy based on the exchange of PECS images to compose a sentence [14]. PECS is used to remedy the discrepancy in communication methods of people with deafness and one or more intellectual disabilities (mild or moderate) who communicate through SL and those who do not use it [15]. In this way, the studies, proposing illustrated dictionaries to facilitate the writing of messages intended for their communication partners, have shown that, with adequate preparation for the rules of use, the subjects have been able to successfully use these tools and have had the role of facilitating writing [16].

In this perspective, we propose the use of a way of communicating with short and silent moving images, the Graphics Interchange Format (GIF) [17] to facilitate written communication via chat, according to the representational assisted AAC methodology. The prototype (better described in [11]), as mentioned, has a set of GIFs representing sentences and words expressed in SL that can be selected and sent to the recipient by the user. This set, although limited, is a representative example of the proposed methodology for chat communication of deaf and non-deaf people. Further studies are needed to extend the basic idea to the various existing sign languages (and will take into account the interface localization so that the gif gallery is shown in the selected language) and to overcome the limitations described above as well.

Fig. 3.
figure 3

Main components of LISA chat tool

Full size image

4.4 Operator and Interpreter Functions

The operator and interpreter interfaces have been implemented in a user-like way:

  • the operator can use the same functions available for the user role. During the registration phase, however, the user is asked to indicate the affiliation, in addition to personal data. The PA operator can search for an “available interpreter” for an immediate call. This is necessary when a deaf citizen comes to the office at any time without assistance.

  • The interpreter cannot search for and book other interpreters. An interpreter can set its immediate or future availability (to be booked for an appointment). The “immediate mode” allows to receive video calls without an appointment. An interpreter can also manage their own appointments (accept, decline, and simply get information).

5 LISA Architecture

The structure of the LISA PWA has the same three tiers of a web application, with the addition of two elements: The Manifest and the Service worker (Fig. 4).

Fig. 4.
figure 4

LISA Architecture showing the interconnections between the five tiers

Full size image
  1. 1.

    Presentation tier: it is the user interface of the web application; the technologies used are HTML5, CSS3, JavaScript and its jQuery library; and Bootstrap, which contains design patterns based on HTML, CSS, and JavaScript, used for the various components of the interface.

  2. 2.

    Logic tier: this is the application processing component. The main technology used is the PHP programming language.

  3. 3.

    Data tier: it allows you to manage and shape the information content of the app; the technologies used, contained in the WAMP multiplatform, are MYSQL database and Apache.

To make LISA tool a progressive web app (PWA), the Manifest and the Service Worker components are required:

  • The Manifest is a JSON file that describes to the browser the application structure and elements, and the tasks to carry out when the app will be installed on the home screen of a mobile device. So, the manifest includes information such as the name of the app, the icons to use to install it, and the main address to start it.

  • The Service Worker is a JavaScript file performed by the browser in the background, which performs background synchronization and network request management operations via the cache.

6 First Evaluation

The design of the prototype was based on requests and suggestions provided by a group of people from the local Deaf Association. The same group evaluated the first app prototype. The evaluation tests focused on the functions developed (registration, interpreter booking for an appointment, video call). In particular, the people involved have appreciated the new AAC-based system proposed for short messaging (chat), as it takes into account the needs of those who have difficulty with written texts and therefore prefer to communicate with SL. So, even though the chat system offers simple words and phrases, the idea was assessed positively. As for the functions of the prototype, users stated that a completely remote connection for the three people would be very interesting.

7 Conclusions and Future Work

In this work, we present a possible application that supports communication between the deaf citizen who uses SL to communicate and the hearing person who uses natural language. The proposed solution aims to promote communication and inclusion of the deaf person in everyday life scenarios, through: 1) a remote SLIS (videoconference) both instant and on demand (booking), and 2) written communication through an assisted chat system accessible and usable by the community of people who use SL.

Starting from previous research on possible ICT solutions for overcoming the communication barriers existing between deaf and hearing people, this work aims at providing a further step forward in the field by proposing the LISA tool.

The LISA tool was developed as a Progressive Web App (PWA), to offer a multi-device service that facilitates communication between PA and deaf citizens. However, although the context considered in this work is related to PAs, the proposed application can be adapted to many other contexts where communication between deaf signers and hearing people is needed. The PA has been selected as a potential scenario because the local Association for the Deaf suggested this use of context. However, the application can also be used to enhance private and personal contexts.

The GIF-based chat messaging service was really appreciated by the users involved in the evaluation. We have proposed an AAC-based system for writing very short messages accessible to deaf people who have problems with written texts and rely mainly on sign language communication.

Future work will include further development of the gateway-based chat system for automatic translation between chats from written text to GIF in LIS and vice versa, to make written communication between a deaf person and an auditory interlocutor more effective and satisfying. This can be implemented for simple text but could be interesting for more complex communications. Another possible improvement could affect the booking functionality: the deaf user could book an appointment with both the PA operator and the SL interpreter, and possibly carry out the videoconference completely remotely, as suggested by some users during the evaluation.