Keywords

1 Introduction

Touch is an integral part of our perception of the world. Touch allows us to manipulate and interact with our surroundings, conveying not only sensations, but environmental information such as the size and shape of the objects our environment is made of [1]. Touchscreens are revolutionizing how we communicate and interact with each other and with the virtual world. However, commercial touchscreens are still dependent on visual and auditory forms of communication, leaving “touch” capabilities largely unexploited. This leaves the platforms lacking in universal accessibility. With the recent incorporation of haptic (vibratory) feedback into touchscreens, a more universal, accessible experience is being realized.

Because of its potential, there has been a surge of interest in developing enhanced haptic interactions with touchscreens. Haptic feedback has been shown to improve users’ ability to navigate a screen [2, 3], and enable users to complete tasks more efficiently [4, 5]. Numerous efforts (e.g. GraVVITAS [6], TPad [7], and TeslaTouch [8]) have made great advancements in making accessible haptic interaction a reality. In comparison to vibrotactile displays, line following on electrostatic surfaces is quite challenging, as feedback is motion-dependent [9, 10]. Vibrotactile displays tend to be more successful for edge detection in regard to graphics such as maps and graphs, despite needing a thick line width and having an element of imprecision for jagged lines [9, 11, 12]. This underscores the importance of understanding the possibilities of vibrations, particularly with the advent of other surface haptic capabilities evolving on touchscreens.

With greater hardware capabilities being realized, it has become necessary to develop an understanding of how to appropriately leverage this vibrotactile feedback to complement the visual interface and enhance its ease of use and accessibility. This research explores potential solutions to a pertinent challenge concerning current touchscreens: non-visual interpretation of graphics. We approach this by investigating a shape identification task using vibratory feedback. Basic shapes were chosen, as they are often the make-up of larger, more complex graphics. The results of this shape exploration task will enable us to obtain a greater understanding of the feasibility of non-visual exploration of graphics via haptic feedback, serving to inform the design of future multimodal interfaces.

2 Background

Designing a system for non-visual interpretation of graphical components on vibratory touchscreens is a challenging task. As with sight, a user must be able to differentiate between the distinct components of a shape, such as vertex points and lines. Additionally, a user must be able to trace the given shape reliably using vibrations. Finally, a user must be able to formulate a mental understanding of the shape and identify it. This process requires a large cognitive component.

2.1 Perceptual Processing

While vibratory touchscreens have opened up new pathways of information transfer, there are limitations in the resolution and perceptual processing of vibrations. Vibrations stimulate the entire fingertip, making finely detailed information that would be easily discernible physically and visually quite difficult to distinguish on a smooth tablet. Moreover, constant vibration can lead to sensory fatigue, limiting a user’s ability to perceive different vibrations over time [13].

Another major challenge is that tasks on a touchscreen (e.g. understanding graphics, especially spatial graphics such as maps) are cognitively demanding when performed without visual assistance. Although non-visual graphics interpretation is possible, it can take users much longer to understand and interpret visual information as conveyed by vibrations on a touchscreen [6, 12]. However, there is evidence that perceptual learning that normally occurs in vision may also occur in touch, as more experience with haptic exploration can lead to fewer judgement mistakes [14]. The ability to create mental representations of otherwise visual items as conveyed by vibrations is extremely important to the success of any developed set of non-visual graphic guidelines. In this work, we contribute to our understanding of using vibrations to represent graphical components through exploring two-dimensional, geometric shapes.

Another challenge exists in the replication of important physical cues from 3D objects which cannot be easily represented on commercially available touchscreens [10]. When exploring 3D objects, individuals have the added benefit of being able to judge an object by size, weight, and texture [15, 16]. When exploring raised line graphics, individuals can judge the direction of continuous line segments and easily differentiate points that are very close together [10]. To understand objects and raised line drawings, individuals can then use strategies such as expanding the haptic field of exploration (using the entire arm instead of just one finger) [14, 17] and sweeping (of the hand/fingers) over a small surface to create a better understanding of the overall shape and its texture [18–20]. Leading an individual’s finger to trace a raised line drawing is also a beneficial strategy that promotes recognition of the drawing, although individually tracing a drawing still yields acceptable identification rates [21].

Because they are smooth, flat surfaces, commercially available touchscreens are not capable of conveying the physical characteristics described above, making shape and object identification on touchscreen surfaces quite difficult. Nonetheless, it is necessary to incorporate amodal cues to assist non-visual users in navigating and interacting with a touchscreen display [15]. One way this can be accomplished is by indicating important areas of the figure on the touchscreen [10], which has been incorporated into our current study by using a different vibration pattern for shape vertices than the pattern representing basic shape lines, echoing previous work done by Concu and Marriott [6]. To facilitate shape recognition on a touchscreen, Goncu and Marriott [6] found that providing stronger vibrations or different sounds at vertex points helped the user obtain a mental representation of important shape features, such as line direction. In this work, we seek to discover if this strategy remains suitable for haptic-only exploration of 2D shapes among a larger number of users with an imposed time limit, as no user wants to spend an unusual amount of time trying to identify graphics, especially considering that visual exploration is fairly quick.

Due to the absence of physical cues such as texture and contiguous line direction, it is also beneficial to understand how exploration procedures on a smooth touchscreens surfaces may differ from those employed to explore physical objects and raised line drawings. In this work, we observe the exploratory procedures employed in object identification on touchscreens with built-in vibration capabilities to better understand which methods are most commonly used to explore graphical components without vision and which procedures tend to be most successful, in order to compensate for or promote certain strategies in later research.

Despite these challenges, vibratory feedback is currently the state of the art in touchscreens and recent studies have shown that vibrations can relay semantic information and have promise in the non-visual interpretation of graphics (such as bar charts and spatial maps) [12, 22, 23]. In this work, we seek to answer a more fundamental question exploring how vibratory lines, varying in thickness, can be used to promote the exploration and identification of basic geometries, which are often the building blocks for complex graphics. This research informs our understanding of the possibilities of vibrations, but also their limitations, building upon prior work that has suggested potential, but has not yet been validated in user studies with large sample sizes.

3 Research Questions

The goal of this research is to determine the feasibility of non-visual graphic identification with vibratory feedback while gaining further insight on the procedures employed to explore shapes. We propose two key questions: (1) Can users identify basic shapes without their sight using only vibrations and (2) What are common procedures used to explore shapes non-visually, and what relationship do these procedures have on the successful identification of 2D graphics?

We hypothesize vibrations alone can facilitate shape recognition on touchscreens for users who cannot use their sight to use the touchscreen, for reasons such as low-light conditions and visual-impairment. We also hypothesize that first exploring larger shapes with thicker borders facilitates similar identification results of smaller shapes that have thinner lines (as defined Sect. 4.2). In this work, we collect navigation procedure data from users to discern the most popular procedures for shape identification to inform future research in the area of a pool of procedures of which users may employ and which may aid in successful identification of graphic elements. Answers to these inquiries will provide a better understanding of how users perform graphic identification tasks non-visually and will also help inform designs of future non-visual touchscreen interfaces, providing necessary end user insights that can inform both hardware and software design for future platforms.

4 Methods

4.1 Participants

Demographics. Fifty-six individuals were recruited from a mid-sized midwestern university and included introductory psychology students, junior-level mechanical engineering students incentivized with extra credit, and volunteers from the campus. Of the engineering junior students, a handful of participants spoke English as a second language, though the exact number was not recorded. One participant was excluded from the analysis pool due to device issues. Male participants accounted for 69.10 % of the total participant pool. The participants reported belonging exclusively to two age groups: 18–20 years (47.30 %) and 21–25 (52.70 %).

All participants were sighted, but blindfolded for the duration of the study. Sighted, blindfolded individuals were chosen to participate in this study due the results of the current study having applications for partial-sight conditions as well as for blind and visually impaired users. The researchers acknowledge that sighted, blindfolded individuals have different perceptual capabilities than visually-impaired individuals [24] and so understand that the results may be difficult to generalize to the visually impaired population. This work should be viewed as an initial usability study.

This study was approved by Saint Louis University’s Institutional Review Board and participants were provided informed consent.

4.2 Measures and Materials

Shape Exploration and Identification with Vibrating Lines. The purpose of this task is to measure the participant’s ability to trace and identify basic shapes (a circle, square, triangle, and pentagon) within 90 s while blindfolded. This study focused only on shape exploration, however we acknowledge that there are other challenges on touchscreen exploration including line segment following. The 90 s time limit was imposed to encourage users to make decisions about shapes within a reasonable amount of time, as would be expected in a true-to-life scenario. Shapes were chosen for their perceived complexity according to the number of vertices found on the shape.

In this task, users were given 8 shapes to explore. Users were told there would be four large followed by four small shapes. This order was imposed to observe the effect previous experience has on smaller versions of the aforementioned shapes. Large and small shapes were not presented in the same order, to encourage users to explore the shape before making a guess as to what it was. Users were not told what shapes would be presented, given options from which to guess, or told what procedures to use. This was done to determine if priming is necessary for non-visual exploration of the interface, or if natural procedures are sufficient.

Large shapes were roughly 70–83 mm in diameter with 8 mm thick vibrating outlines. Small shapes were roughly 26–34 mm in diameter with 4 mm thick vibrating outlines, about half the size of large shapes (Fig. 1). Giudice and colleagues [12] used line widths of 8.9 mm in similar tracing tasks (geometric shapes, bar graphs, and letters) and found information interpretation via vibration on a touchscreen to be comparable to hardcopy interpretation, though learning time was markedly longer. In this work, we wanted to decrease the size and line widths of the shapes to observe how such a change might impact identification accuracy and exploration procedure within a large sample size.

Vibration patterns were taken from Immersion’s Universal Haptic Layer (UHL) library of vibrations [25]. Shapes used the UHL vibration Short Buzz at 100 % power to denote line segments and Buzz with Bump at 66 % power to denote vertices. The line segment vibration was chosen for its unobtrusive quality and the vertex vibration was chosen for its similarity to the line vibration, but with a bump every 250 ms, making it feel stronger than the line vibration.

Fig. 1.
figure 1

Size comparison of the large and small triangle.

Full size image

4.3 Procedure

User trials took on average 30 min to complete. After completing a demographics form, users completed the Shape Identification (SI) task on a \(10.1^{\prime \prime }\) Samsung Galaxy Note. This tablet remained stationary on a table. Users were blindfolded for the duration of the SI task.

For the SI task, users were given up to 90 s to determine each of the 8 shapes. Again, the 90 s time limit was imposed to encourage users to make decisions about shapes within a reasonable amount of time. They could explore the shape with only the finger of their dominant hand. While users explored the shape, the researcher categorized the user’s finger movements. The same researcher categorized all participants, as they were the only researcher on-site for the study. When users felt they knew the shape, they could ask the experimenter if their judgment was correct. Users were not given any options to choose from. If correct, their time was recorded for the shape and they were presented with the next shape. If incorrect, users were able to continue to explore the shape until their 90 s expired and could make one more guess.

We did not want the time users would take to find the shape to interfere in this task as our concern was only with shape identification and exploration, not navigating to it. Therefore, each participant was started on the top, centermost part of each shape’s outline. Upon the completion of the SI task, the study concluded.

Exploration Procedure Classification. One researcher conducted all of the user studies and classified each of the users’ procedures on every shape explored. Procedures are based off of Lederman and Klatzky’s [20] hand movement classification, specifically “lateral motion” for its relevance to touchscreen navigation. In the current study, two qualities are derived from the lateral motion movement: (1) speed, and (2) goal-orientation. The speed quality is self-explanatory, while the goal-orientation quality is further categorized into the following: (1) deliberate movement, indicative of careful exploration; and (2) sweeping movement, indicative of trying to understand as much of the shape in as little time as possible. Only one rater was used due to availability on-site. Classifications are described in further detail below.

Slow and Deliberate Movement: Slow finger movement with very few deviations from the vibrating outline.

Slow and Sweeping Movement: Slow finger movement with many deviations from the vibrating outline in a systematic manner.

Quick and Deliberate Movement: Quick finger movement with few deviations from the vibrating outline.

Quick and Sweeping Movement: Quick finger movement with many deviations from the vibrating outline in a systematic manner.

5 Results

In the above study, we explored two fundamental questions: (1) Can users identify basic shapes without their sight using only vibrations and (2) What are the kinds of procedures that are most commonly employed by users exploring shape components non-visually, and how do these procedures affect shape identification?

Cronbach’s Alpha was used as a measure of internal consistency and was found to be good for both measures of shape identification time and shape identification accuracy (\(\alpha \) = .62; \(\alpha \) = .42). Scores were fairly consistent with each other. Average time and average accuracy were both normally distributed.

Table 1. Shape identification accuracy of large (L) and small (S) shapes
Full size table

Most participants were correctly able to identify the shape by its vibration (Table 1). Smaller shapes tended to have slightly higher accuracy than their larger counterparts, although both the small (48.1 %) and large (46.6 %) pentagon were particularly troublesome to determine. A negative correlation was found between shape identification time and shape identification accuracy (r(53) = \(-.61\), p < .01), implying that slower times were associated with lower percent shape identification accuracy.

Table 2. Identification procedure frequencies of large (L) and small (S) shapes
Full size table

Identification accuracy was related to observed procedure use, although identification time was not. This is consistent with the experimenter’s observations that the longer individuals spent trying to explore the shape, the more likely it was for them to be incorrect. In Table 2, the procedures are as follows: Proc. 1, slow and deliberate movements; Proc. 2, slow and sweeping movements; Proc. 3 quick and deliberate movements; and Proc. 4 quick and sweeping movements.

Overall, high accuracy was related to low use of quick and sweeping movements (r(53) = \(-.29\), p < .05) and high use of slow and deliberate movements (r(53) = .34, p < .05). We observed that the majority of the time, participants used a movement procedure that could be classified as slow and deliberate, although this effect was not as dominant for smaller shapes. For smaller shapes, participants began to consider Proc. 2 (slow and sweeping) and Proc. 4 (quick and sweeping) as viable exploration procedures.

6 Discussion

Our hypothesis regarding shape identification by vibration was correct in that the shapes could be reliably identified by their vibrating boundaries (>78 %), with the exception of the pentagons. We attribute this to the number of vertices the pentagon has and the effort involved in making a complete mental representation of the pentagon shape. We also attribute this to the fact that participants may be less familiar with pentagons, as they are encountered less frequently in the environment than the other shapes, thus participants may have had been unable to recall the exact name of the figure.

In line with our expectations, smaller shape sizes were slightly more accurately identified than larger shape sizes, indicating that, when given the opportunity to practice on larger shapes, smaller size is not as detrimental to shape identification accuracy as expected. This finding informs us that simple, 2D shapes may not have to be abnormally large to be accurately interpreted in virtual space with vibratory feedback. This finding is encouraging in regards to the design of non-visual user interfaces and the interpretation of graphical data and charts. This echoes previous findings, in particular [12] and [23], but adds a size dimension to the research.

Fig. 2.
figure 2

Four cases of unexpected exploration procedures (a) crossing the vertices, (b) circling at vertices, (c) broad scanning, and (d) short scanning.

Full size image

If the proper shape could not be identified by the discovery of the second or third vertex point, many users could not successfully identify the shape, indicating the importance of a procedure that quickly finds key shape points. It was easy for users to miss key vertices when quickly exploring the pentagons. This may indicate that a better strategy is necessary to highlight key components of more complex shapes, such as allowing the use of physical reference points on the screen (such as using another finger to keep track of an important area).

In general, slow and deliberate seems to be a good procedure for beginning exploration, being employed the most on large shapes then less frequently on smaller shapes as users began to seek an alternative procedure for smaller shapes (see Table 2). Slow and deliberate use begins to decrease possibly as users become more aware of the key features to look for (vertices, straight lines).

The four categories of procedures were not all-encompassing, though they were descriptive enough for the majority of the user movements in the study. Only a very small handful of individuals employed creative procedures that we did not expect to see and some are displayed in Fig. 2. The above users (a) made crosses at vertex points in order to determine the next line segment (correctly identified); (b) circled the vertex points to find nearby line segments (correct); (c) “scanned” the screen with lines (correct); and (d) another scanning procedure variation with shorter lines (incorrect). As we have limited data on these interesting procedures, we cannot make any comments about their possible effectiveness.

Fig. 3.
figure 3

Density of finger positions showing the areas most explored by users in each of the small and large shapes. (a) Triangles; (b) Circles; (c) Squares; and (d) Pentagons. (Color figure online)

Full size image

Figure 3 shows the finger position of each shape aggregated from every user. Vibrations could be felt inside of the white lines. Warmer colors indicate a higher frequency of finger positions in that area. In general, users did not deviate far from their starting position and nearby vertices. Generally, users were confident in what shape they were exploring by the second or third vertex point, most visibly illustrated by the densities of the triangles and squares.

We observed that vibration discrimination of vertices and edges can be very difficult if the participant is moving too fast trying to explore the shape. This was extremely apparent in both pentagon trials, as users tended to guess that they were circles if they tried to explore the pentagons too quickly (missing key vertex points). Users who moved too quickly also reported feeling “phantom vertices.” Phantom vertices provided “fake” mental reference points which hindered further exploration of the shape. This was most commonly reported on the circles which have no vertices. For some users, vertex vibrations became almost impossible to distinguish from normal vibrations due to sensory desensitization.

In closing, we have learned that basic shapes can be identified by vibratory feedback alone with >78 % accuracy, excluding the complex pentagons which could be identified <50 % of the time. The mechanism of indicating vertices with a stronger vibration than the rest of the shape yields satisfactory results until the shape becomes too complex with too many vertices.

7 Conclusion

In this work, we have expanded the current body of research on how non-visual graphics interpretation on touchscreens. Our results demonstrate that haptic lines do not need to be unnecessarily large or thick to facilitate mental representation of spatial information, but that a learning or training period may initially be necessary. Our results also show that slow, deliberate tracing exploration movements are more successful than sweeping movements, at least until the individual becomes aware of how to identify key features in the graphic.

Our findings will lead to new usability features developed based on better understanding of spatial and geometric information as conveyed through vibration. This will likely have applications for blind and visually impaired users who cannot rely on vision to use touchscreen devices. These findings may also benefit situations where individuals cannot rely on visual or auditory information for fear of putting their safety at risk, such as soilders in the field.

Line following via vibrations still remains a challenging part of non-visual touchscreen navigation. Future work will focus on developing specific procedures to facilitate better line following ability and quicker 2D object recognition. Future work in this area will lead to universal design guidelines for an effective, accessible, multimodal touchscreen interface that is not limited to visual interactions.