Keywords

1 Introduction

The graphical contents that comprise images and maps are effective in transmitting a considerable amount of information in a small space, allowing better organization of series of data and information so that they become easily understandable. They are widely used in the teaching process at all levels of education. The sense of vision enables a comprehensive and quick understanding of graphic content. While visually impaired students do not have easy access to this type of content, the use of other sensory subsystems, such as hearing and haptics (touch and kinesthetic), can help in the interpretation of these visual contents.

Tactile graphics are mainly used to help visually impaired students explore graphic representations. To access tactile graphics, the visually impaired (VI) require the support of a person with vision (seeing person) to provide the initial information such as the general message transmitted by the tactile drawing and to guide the individual in the step-by-step reading of the drawing [7]. The absence of this person makes it impossible for the VI to use tactile graphics [16].

Full sound description of the content is an alternative to tactile exploration to access graphic content, and audio is preferred in tasks of exploration and navigation [11]. However, studies showed that the VI tend to prefer tactile presentations to audio exclusively. This shows that the combination of responses of various sensory systems–for example, by combining a tactile sensory response for graph exploration and an auditory response for obtaining complementary information–can have a greater effect on the transmission of graphical information to the VI.

The Brazilian legislation, since the constitution of 1988 [4], established in its articles 205 and 206 that education is a universal right, and that education must be provided based on the principle of equality of conditions for access and permanence in schools. The Law of Guidelines and Bases of National Education, published in 1996 [5], in its article 59, agrees with the view of the constitution on the right to education of disabled people. It also states that education systems must provide specific curricula, methods, techniques, educational resources, and organization to learners with special needs in order to satisfy their needs.

In fact, the entire Brazilian legislation on the inclusion of disabled people is relatively ambitious regarding the promotion of inclusion. However, institutes face many difficulties, especially regarding the budget and the training of personnel to support disabled students and to provide them with the necessary support for good educational development. Some authors state that “there is a contradiction between speech and reality, because the visually impaired, who depend on public resources for their education, face many barriers and challenges” [14].

Assistive technologies (AT) “are resources that enhance the functional abilities of disabled people” [3] and, therefore, can be used to fill the gaps in teacher training and the lack of specialized material. AT can help in several challenges imposed by the presentation of graphic content, helping in the development of correctly prepared tactile graphics and enabling the VI to read tactile graphics on their own. Thus, this work presents an application for tablets designed to help the VI in reading tactile graphics. After studying different strategies for the exploration of tactile graphics adopted by VI people, we propose an application that utilizes the benefit of multimodal interactions. A printed tactile graphic is superimposed on the screen of the device. During the tactile exploration of the graphic, the application provides enlightening auditory information about the area explored by the VI through vocal synthesis. The objective is to supply the assistance given by a seeing person, providing full autonomy to the VI in the reading and interpretation of tactile graphics. The interactions were implemented in order not to impact the strategies of tactile exploration of the graphic.

2 Related Works

Several solutions and approaches can help the VI in the exploration and interpretation of graphic content. First, we can cite the main items presented by related works: TTT (Talking Tactile Tablet) [13] and IVEO Touchpad [8]. Both solutions allow the overlap of tactile graphics on a touch-sensitive hardware connected to a personal computer, via USB, that runs the software allowing the tactile graphics to become interactive. Touching the tactile graphic triggers audio information, which allows more information to be transmitted to the VI.

Although the connection to a computer provides TTT and IVEO Touchpad with considerable processing and flexibility, it limits its portability as the device needs to be physically connected. However, the main limitation of these devices is that they rely on specific hardware, which is not only bulky, but also used exclusively for a certain task (reading tactile graphics printed and overlapped on a touch-sensitive hardware). Another aspect to be highlighted is the high commercial value of these devices, which makes them unacceptable in terms of economic realities with no considerable investment in accessibility for people with disabilities (Fig. 1).

Fig. 1.
figure 1

a) IVEO Touchpad (Source: reproduction/viewplus.com). b) Talking Tactile Tablet (Source: reproduction/ITD Journal

Full size image

In contrast to the use of specific hardware, we can find solutions that aim at helping the VI in the task of reading and interpreting tactile graphics and that are based on common hardware, i.e., the hardware can be used to perform other tasks such as Internet browsing, reading books, and accessing multimedia content. Two of these solutions are GraVVITAS [10] and Figure Perceiving Tools [6]. Both solutions allow the VI to access graphic content using a touch screen tablet and can provide audio and haptic feedback of the graphic information displayed on the screen.

There is a small difference between these two solutions on the generation of haptic feedbacks. In GraVVITAS, these feedbacks are provided through vibration motors placed in a glove designed specifically for this purpose. Whenever the user touches any graphic object on the touch screen, the system provides haptic feedback by activating the vibration motor in the glove. The Figure Perceiving Tools, in contrast, do not require the use of a glove. Haptic feedbacks are provided by the vibration of the tablet itself (Fig. 2).

Fig. 2.
figure 2

(a) GraVVITAS (Source: reproduction/[10]). b) Figure Perceiving Tools (Source: reproduction/[6]

Full size image

However, in a tactile exploration with both hands and several fingers, it is difficult to distinguish at which point the graphic object displayed on the screen is being touched. This prevents tactile exploration with both hands and multiple fingers, which is not a natural way for the VI to explore tactile graphics [1, 11]. Another aspect hinders the tactile interpretation of the graphic through vibrations only: the finger simultaneously perceives several points of the printed tactile graphic, which gives a notion of continuity and direction of lines and intersections and allows differentiating lines by size and areas by textures. Vibrations only provide information regarding the presence or absence of the finger at a point and, even if we imagine semantics of the design of vibration that allows differentiating lines or areas (by the intensity of the vibration), it appears challenging to transcribe the perception of continuity or intersections only through this feedback.

3 Comments on Tactile Exploration Without Technological Resources

To develop a solution that satisfies the real needs of a VI student, it is necessary to consider a participatory and user-centered design, because “this approach prevents an engineering trap, with development being driven by computational efficiency and assumptions of designers, often wrong, without considering feedback from the end user” [9]. For this purpose, three observational studies were conducted to analyze the strategies of the VI individual used to explore the graphic content printed in relief.

These studies were conducted during real classes with the VI accessing graphic content (as shown in Fig. 3). One student was a high-school entrant, one student was a high-school graduate, and the third was an engineering graduate. Consequently, the three individuals had very different experiences with tactile graphics. The first admitted having had little contact with this resource owing to the lack of adapted material during his elementary school. The other two students were more familiar because they had attended institutions that sought to use this resource in the best possible way.

Fig. 3.
figure 3

Observation of how the VI explore graphic content with the help of teachers during the classes

Full size image

At this stage, we sought to identify: (i) how the VI explores the tactile graphic (use of hands, successive stages of exploration), (ii) what information is provided by the seeing person, and (iii) when and why these descriptions are provided to the VI. The main points observed in this study are highlighted below.

  1. a)

    In the initial phase of tactile exploration, teachers provided general information on the tactile drawing, explaining the main subject addressed and the spatial organization of the content. At the same time and/or in sequence, the VI devoted time to a general exploration of the tactile drawing. This first general stage of “recognition” was performed with one or two hands. The use of one or two hands appears to be related to the level of experience of the student with tactile graphics;

  2. b)

    During the tactile exploration, the students were guided by the seeing persons (teachers) on how to explore the tactile drawings and which way to go, with many of them directing the students’ hands;

  3. c)

    As the students explored the tactile image, the seeing person informed them about the explored area and the VI could simultaneously explore the tactile graphic details of that given area.

  4. d)

    In the second stage of graphic exploration, the participants tended to continue using one or two hands according to the strategy used in the first stage. In some moments, one hand served as a spatial reference for measuring the distance between the objects.

Here, we highlight some prerequisites to an AT to be naturally integrated in the exploration of tactile graphics by the VI without requiring that they change their methodology of work. Thus, the technology must: (i) be resilient to interaction with both hands; (ii) provide general information on the graphic at the beginning or when requested by the VI; and (iii) allow the VI to request complementary information on the graphic without the information polluting the printed graphic. The observation also revealed that the directing of the VI’s hand during exploration could be relevant in certain circumstances, but further evaluations are required. This feature would hardly be enabled without the use of a specific device. Another point observed during these studies, and confirmed by one of the participants, is that tactile drawings with many subtitles and many details can complicate tactile exploration, and hence, it is recommended that everything is well organized and with as few subtitles and details as possible, so that the VI can understand the drawing more quickly and effectively. Some authors agree with this observation [12, 15].

4 Application Development

T-TÁTIL involves the use of a common tablet with an application installed to detect the gestures of tactile exploration by the VI (Fig. 4.a), with a printed tactile drawing positioned on the screen of the device (Fig. 4.b) allowing conventional tactile graphics (without audio) to become multimodal tactile graphics with sound feedback, which connect sound and touch to improve the exploration of the tactile drawing (Fig. 4.c).

Fig. 4.
figure 4

Application functioning model

Full size image

The application associates the printed tactile drawing (Fig. 5.a) to a previously created file in SVG format. The SVG contains forms and textual content associated with forms that describe the areas of interest of tactile drawing (These areas and descriptions are inherent to each drawing). The shapes drawn in the SVG file are converted to clickable areas in the application interface (Fig. 5.b). Thus, during the tactile exploration of the graphic superimposed on the tablet (Fig. 5.c), the user can activate the sound descriptions to request additional information. The text content associated with the activated area is read through voice synthesis. A general description can be accessed by clicking an area located in the lower right corner of the interface.

Fig. 5.
figure 5

Use of T-TÁTIL

Full size image

4.1 Technical Considerations of the Project

The use of touch screen devices was essential for the development of this solution. This technology allows the interaction of users with electronic systems only using a touch screen, activated either by the finger or a special pen, without requiring the use of another peripheral device such as a mouse and keyboard. Currently, two types of touchscreens are most commonly found in devices on the market: resistive screens and capacitive screens. Capacitive screens operate from an electrically charged capacitive layer positioned over the screen. By touching the screen, some of these electrons are transmitted to the finger, and hence, the device understands this small discharge of electricity, calculates the coordinates and then translates it to a command, either a touch or a specific gesture. The resistive screens, in contrast, function by means of pressure exerted on the screen that changes the electric field and allows identifying the location of the touch or gesture on the screen (Fig. 6).

Fig. 6.
figure 6

Representation of capacitive screens (left) and resistive screens (right) (Images: reproduction/Techtudo).

Full size image

Based on this information, it was necessary to test whether capacitive touchscreens can detect gestures and touches even with the paper positioned on the screen of the device. This problem does not exist a priori in devices that use resistive screens because the operating principle is the pressure exerted on the screen. However, in capacitive screens, which are most widely used, a printed tactile graphic could act as an insulator and prevent the transfer of the electric charge.

To validate the functioning of capacitive and resistive screens, a small experiment was conducted, in which a printed tactile graphic was superimposed on the touch screen of several devicesFootnote 1 found in local institutions and shops in the region. As illustrated in Fig. 7, the Paint Free applicationFootnote 2 was used to determine whether finger pressures were detected through the paper . The lines of each shape of the drawing were followed only once (Fig. 7.a). The experiment was conducted with Braille paper and flexi paper, both easily found in Brazil.

Fig. 7.
figure 7

Touch recognition test with overlapping tactile drawings on the screen of the device

Full size image

The results confirmed that, regardless of the type of paper used and the device tested, the drawings made in the “paint free” tool using the tool in conjunction with the paper were correctly reproduced. With the “undo” functionality, it can be observed that there was no discontinuity in the interaction (Fig. 7.b). Consequently, we infer that the touches and gestures performed were correctly detected through the papers.

4.2 Paradigm of Interaction Resilient to the VI’s Methodology of Work

From the observational studies mentioned above, it was possible to identify several forms and methodologies of tactile exploration by VI people. Thus, it is possible to define an interaction model for the application that does not harm such methodologies and thus interferes as little as possible in the form of tactile exploration of each user of the application.

  1. a)

    We observed that the VI individuals used several fingers when performing tactile exploration, even using both hands simultaneously, and rarely using only one finger. This considerably complicates the detection of the area of interest that could be used to play the sound information automatically. Thus, the audio description must be activated by demand, i.e., when the VI request it through a specific interaction (for this application, we have implemented the double touch functionality in the area of interest).

  2. b)

    Similarly, there must be a way to interrupt the sound descriptions if the user wishes. To this end, we have implemented a simple touch screen interaction that immediately ceases the description being reproduced. This type of interaction does not hinder tactile exploration because it generates sliding events with one or several pointers on the screen, and not simple touch events. Further updates will allow for improvements, as demonstrated in the discussion and future works sections.

5 Case Studies

5.1 Method

To assess whether VI people can use the proposed approach, and to obtain qualitative feedback, we conducted a study with end users of the application. To evaluate the usability of the application, a low-cost Android tablet (Positivo “YPY-AB10E with 10” screen) was used with the tactile application installed.

With the help of a professional with vast experience in the elaboration of adapted materials, currently working at the Federal Institute of Education, Science, and Technology of Pará with more than 10 years of experience in supporting students with special needs during academic life, we projected seven tactile drawings in several categories of teaching: a line graph (LG), a shark (S) and its main characteristics, a genogram (GN), a representation of the Holy Supper (HS), a floor plan (FP), a soccer field (SF), and a map of Brazil (MB) divided into regions. An extra element that allowed us to include a general description of the image was added to all drawings. This element is represented by the symbol of “play” (an arrow pointed to the right direction), which is in the lower right corner of all drawings. Each participant was informed of the existence of this element and that it could be triggered while the tactile drawing was explored.

The files can be downloaded at https://drive.google.com/drive/folders/1nEEpHeW5Vd9OsTNXh8PPaaOTqsftiADK?usp=sharing.

Three VI individuals were recruited to participate in this case study; details of the participants can be found in the next section. Each study session lasted approximately 2 h. Initially, each participant received a brief explanation about how the application worked, how to obtain the sound feedback from the clickable areas, and how to interrupt the descriptions if they desired. They then had the opportunity to explore a test drawing and were instructed to perform all tactile exploration tasks and obtain the sound feedback from the several clickable areas (Fig. 8).

Fig. 8.
figure 8

Testing the prototype of the application with the three participants

Full size image

After this introductory stage, the testing of the prototype of the application started. Each participant had as much time as they deemed necessary for tactile exploration of the drawing and to obtain the sound feedback from the clickable areas. After exploring each image, as soon as the VI reported having understood the drawing, a questionnaire was provided to determine whether the participant could fully understand the tactile drawing. In case of doubt, the VI could query the graphic before responding. These questionnaires helped support the qualitative results of this study and provided some quantitative results too.

Immediately after exploring and responding to the questionnaires, the participants received a second questionnaire that surveyed qualitative data including the participants’ opinions about the experiment, tactile drawings, application, and use of technology.

The questionnaires can be accessed at https://drive.google.com/file/d/1Ys5D9CCF_SlIIbPh4Ei3v9RCdskh9H2z/view?usp=sharing.

5.2 Description of Participants

The first participant, P1, is a student at a high school integrated with technical education in information technology (22 years old). He has been completely blind since he was 3 years old. The participant reported having average braille and tactile drawing skills. He uses braille material every day at school and tactile drawings 2–3 times a week. The participant reported having a good experience using smartphones and computers every day. When asked if he has ever used any technological resource to access graphic content, the participant informed that he uses only tactile drawings printed on braille paper or flexi paper.

The second participant, P2, is a student who has just finished elementary school (15 years old). He has been completely blind since birth. The participant reported having average braille reading skill. He uses braille material 2–3 times a week. He reported that he had almost no skills in tactile drawing because he had used this resource rarely during his school education because according the participant “(Tactile drawings) were not worked on in the municipality where I live.” The participant reported that he had no experience at all in using mobile devices and that he had attempted to use a computer and smartphone before but without success. When asked if he has ever used any technological resource to access graphic content, the participant informed that he uses only tactile drawings printed on braille paper or flexi paper.

The last participant, P3, is a sanitary engineering student (25 years old). He has been completely blind since he was 7 years; however, he was born with a disability in one eye and had only partial vision in the other. The participant reported that he has good braille reading and tactile drawing skills, and that he uses these resources every day at the university. The participant reported having significant experience using smartphones and computers every day. When asked if he had used any technological resource to access graphic content, the participant informed that he uses smartphones to access graphic content such as pictures, photos, and audio description, in addition to tactile drawings printed on braille paper or flexi papers.

5.3 Qualitative Results

Opinion on Tactile Drawings. All the participants indicated that tactile graphics are very important for the teaching and learning process in schools. P3 said that “With tactile drawings, it is easy to understand the relationships between objects and figures, which allows you to create this relationship mentally.” The participants said that the tactile drawings used in the experiments were easy to understand and that the textures and organization were well ordered so as not to confuse tactile exploration. When asked if they had preference for some of the drawings, P1 indicated the “genogram,” the “floor plan”, and the “Last Supper.” P2 liked the “shark” and P3 the “genogram.”

Opinion on T-TÁTIL. All three participants agreed that T-TÁTIL could help in the exploration and understanding of tactile drawings, making them faster to understand and better organized as fewer Braille subtitles are required to explain the details of each drawing. Regarding one of the drawings, P1 said that “it is even faster to obtain the information from the drawing and if subtitles were added, the presentation of this information would require many pages.” The participants said that they would like to use the system mainly for activities at school, during class, and some activities at home.

The participant P3 raised a point of interest regarding this technology; according to him, “the blind community, today, talks about a future problem, that is to pay more attention to virtual technologies and forget physical technologies,” and hence, it is interesting that an application integrates physical elements with virtual elements, and “this project does the opposite. It encourages the visually impaired to have physical accessibility through the analysis of tactile drawings, combining information technology with the use of a tablet.”

Use of the Application. Most of the participants reported not having faced major difficulties in the use of the application, as a few different functionalities had been implemented in order not to confuse the participants in this initial test. We could evaluate the usability and acceptability of the T-TÁTIL.

Although a few instructions are required to use the application, P2 had some difficulties in triggering and stopping the sound descriptions with the gestures provided in the introduction of the tests. We believe that this is due to the participant’s limited experience in using mobile devices and computers.

Errors Verified. During the case studies conducted, it was possible to observe some errors and opportunities for improvements to be implemented in the next updates of the application. The interaction model chosen for the application, which was believed not to have major problems, ended up revealing that modifications need to be made as, for example, when the element of general description of the drawings was activated and a general exploration was performed by the VI individuals, the application sometimes detected a false simple click and interrupted the sound description. Although this problem occurred sometimes, after a certain period of adaptation, the VI participants could trigger the descriptions and explore the drawing simultaneously without the “false single click” being triggered.

Another problem occasionally detected was that, even with the tactile drawings attached to the screen of the device, at times, at the moment of exploration, the paper moved slightly. Consequently, the physical object was no longer perfectly superimposed on the corresponding clickable area, making it difficult to activate the sound descriptions. Another paper positioning mechanism should be implemented to prevent the occurrence of this error in an uncontrolled environment.

5.4 Quantitative Results

The time required for each participant to explore the tactile drawings is illustrated in Fig. 9. It is believed that the reason why P2 required a considerably longer time to explore tactile drawings is because he had no experience in exploring tactile drawings. In general, the time taken by the other two participants varied from one image to another, considering only the time required for the application to reproduce the requested sound descriptions so that the VI could understand the details of the tactile drawing.

Fig. 9.
figure 9

Graph showing the time spent exploring the tactile drawings

Full size image

The number of correct answers to the questions are shown in Fig. 10. We also showed the number of times the VI queried the tactile drawing/application to answer the proposed questions.

We believe that, while answering the questions, most of the queries were made because of the amount of information that was presented in the audio descriptions–for example, for all participants (2 queries for 3 questions), the highest number of queries were made to the map of Brazil drawing divided into regions and including audio information informing which states formed the region, the total area, population, and demographic density. Other drawings commonly queried by P2 were that of the Holy Supper (2/3) and soccer field (2/3). We observed that the queries were usually made for the tactile drawings having more tactile and sound information and this may have required a greater cognitive load, making it difficult to memorize this information completely. P1, in exploring the map of Brazil, said, “I seem to have answered more than three questions. I think it is because of the amount of audio information I had to search for answering the questions.”

Fig. 10.
figure 10

Correct answers and queries for tactile drawings questions

Full size image

6 Discussion

This study supports the claim that tactile drawings are very important for VI students. The lack of use of this type of resource can negatively influence the school education of VI people, making it even more difficult to include them in school curricula. Participants reported that tactile drawings are not always ready in time for use in class and they need support from a seeing person (usually the teacher) to understand the details of the tactile drawing. Hence, in this regard, T-TÁTIL can be even more useful for VI students when they are reviewing tactile drawings after their application in classroom.

This evaluation stage emphasized that, regardless of the tool, the tactile graphic/audio needs to be meticulously developed. Although we observed that the tool helped the VI significantly in the task of tactile exploration, describing the drawing in detail through audio in a timely manner, we realized that, even for more experienced VI individuals, the use of a great amount of sound information, sometimes unnecessary, ends up impairing the understanding slightly. This corroborates the statement of [15] that “it is difficult to add information, such as captions or braille notes, without making a tactile graphic excessively complicated.” The same applies to audio information. This information needs to be clear and directed to the objective of the tactile drawing.

We believe that the solution proposed in this work can help VI individuals at all levels of education and with different levels of skill in braille reading and tactile drawing. This is demonstrated by the participant P2 who, despite having almost no experience in tactile drawing exploration, could accomplish the tasks and obtain satisfactory results in addition to providing positive feedback regarding the solution.

This study aimed at evaluating whether a tablet could help VI individuals in the task of exploring tactile drawings on their own and to establish whether the sound descriptions added to the tactile drawings help in this task. All the participants indicated that T-TÁTIL is useful and provided examples of how and when the application can help in the exploration of tactile drawings. As the prototype has been evaluated and established, further studies evaluating a larger number of participants can be conducted to develop a more complete solution.

6.1 Future Works

In future works, we intend to (i) include other types of sound feedback (e.g., recording of descriptions by a real person, sounds of certain objects or animals, music, etc.) using different gestures for each type of audio feedback; (ii) implement a mechanism that allows coupling the physical chart with its representation in clickable areas on the tablet without having to use menus (initially, the idea is to use QR codes but new possibilities can be analyzed); (iii) create a mechanism for positioning the tactile impression on the tablet screen without the support of a seeing person; initially, we conceived the idea of a base for the tablet with some locks to hold the paper, but there is also the possibility of marking the printed paper so that, as soon as the VI person touches the markings, the application positions the digital file according to these markings; (iv) adjust the form of interaction to interrupt the audio descriptions (simple touch) in order to avoid “false” events generated by tactile exploration; and (v) develop versions for other mobile platforms.

7 Conclusion

The T-TÁTIL was developed by confirming that the participants could interact with the usual screens through the papers used for relief printing, and based on observational studies and related works, both presented throughout this work. The tool, implemented for mobile devices (for now, only mobiles using the Android platform), enhances printed tactile drawings, providing sound descriptions of the drawing according to the desire of the VI.

Thus, we achieved the objective of proposing a solution that helped VI in the reading and interpretation of tactile drawings based on low-cost mobile devices widely used in society. The tool and the services provided by it do not require specific hardware, unlike the alternative solutions proposed in the past. Moreover, the interaction with the device was resilient to the interaction with both hands even though small, simple adjustments can be made to improve this aspect further. Consequently, the tool was naturally integrated by the participants (who were different from each other) into the tactile drawing exploration methodologies without a learning cost.

Although the evaluation performed does not have the statistical significance that allows us to state that the tool, in its current state, provides complete autonomy to the VI in the task of reading and interpreting tactile drawings, the elements raised in this work show that a significant step was taken in this direction.