Keywords

1 Introduction

Haptics is the science that studies biological sensations related to touch. This sensation can be generated from a kinesthetic (force) or cutaneous (tactile) feedback [15]. The haptic sensation can be perceived when a person manipulates a real object or when a user interacts with a virtual object in a computer simulation with haptic feedback [13].

In this way, such a sensation can be created from the use of haptic devices such as a haptic pen [34]; devices attached to users’ fingers that respond to actions and can simulate heat, force, friction, weight and roughness [2, 23, 24]; and even less conventional devices, like a kind of armor attached to parts of the body [1]. These coupled devices can also be used in Virtual Reality (VR) systems, combined with stereoscopic glasses to insert the user into a virtual environment, for interacting with virtual objects [4, 26, 36, 38, 47, 51] and in Augmented Reality (AR) environments, combining virtual and real elements [22].

Haptic sensations can also be created without the use of any physical haptic device or actuator, through only visual changes or distortions. This type of interaction is called pseudo-haptic [32]. When the shape, speed and trajectory of a given virtual object are changed during human-computer interaction, it is possible to generate feelings of friction, mass, stiffness and characteristics of the curvature of the surface. It is possible to change the user’s perception of the shapes of a physical object when the user views changes to a virtual object [5,6,7,8], like the spatial properties of a cursor during the interaction [28, 30, 31].

With the popularization of touchscreen devices, improvements can be studied in user interaction, providing some kind of tactile feedback when performing actions without the use of another physical device. Chubb et al. (2010) [11] called “haptic surface” touchscreens with haptic feedback. The occurrence of haptic sensations on touchscreens can provide a better experience in human-computer interaction, because it is possible to combine the interaction with some feature of the device and, thus, expand the sensations [20]. This type of tactile feedback can be used by modifying the graphical interface, without the need for additional hardware.

Thus, this paper describes a study on the perception of users for pseudo-haptic effect of stiffness in certain materials, rendered graphically on smartphone interfaces, with graphical elements distortions activated through touch, and compared with the same real materials. The goals are to: prove the pseudo-haptic perception generated from purely visual changes; measure the level of perception for different materials; and check the use of the smartphone vibration feature.

The paper is organized as follows: Sect. 2 describes the main concepts for understanding the paper; Sect. 3 presents the related work; Sect. 4 deals with the case study; and Sects. 5, 6 and 7 address the results, discussions and conclusions, respectively.

2 Background

In this section, the concepts of haptic interaction, haptic surfaces and pseudo-haptic interaction that can be used in VR and AR environments to improve the user experience are presented.

2.1 Haptic Interaction

The human being has several sensors spread throughout the body, which allow to perceive the surrounding environment. Through touch it is possible to perceive textures, measure temperature, evaluate a material and identify edges. According to Montague (1986) [39], touch is the first sense to develop. Touch is usually combined with vision and hearing, and it is this coordinated information, coming from different sensors, that allows to expand the sensations and have a better knowledge or perception of the environment.

Haptic is a term derived from the Greek verb “Haptesthai”, meaning “touch”, and refers to the science of feeling and manipulating through touch. At the beginning of the 20th century, this word was introduced by researchers in the field of experimental psychology, referring to the active touch of real objects by human beings. In the 1980s, there was a redefinition of the term to broaden its scope and include all aspects of touch, involving human-computer interaction. Today the term is multidisciplinary, covering areas such as Medicine, Psychology, Computer Science and Engineering, which study the human touch and interactions with computer systems [15].

The haptic sense is very important in people’s daily lives and the exploration of the haptic sensation is often still marginalized in interactions with computer system interfaces, which focus on visual presentation (graphic effects) and audio (sound effects). The most common way to reproduce haptic effects in these systems, which involve VR and AR applications, is through some haptic device, designed to provide effects, such as stiffness, using force or resistance.

2.2 Haptic Surface

The use of haptic content on touchscreens is known as “haptic surface” [11]. This term refers to surfaces that can generate haptic effects on a physical display to stimulate the biological receptors present in the hands. With the increased use of touchscreens for human-computer interaction, solutions that provide the tactile sensation are sought. In this sense, these solutions can combine these screens with certain features, which can be from the most common and known, as a vibration on a smartphone, to something more complex, such as dynamic surfaces.

Chubb et al. (2010) [11] separated the haptic surfaces into three categories as follows: vibrotactile displays; variable friction devices; and shape changing surfaces. Vibrotactile displays are those that use one or more vibration motors that respond to the user’s touch action, such as when tactile contact with a smartphone screen occurs when clicking on a button on a graphical interface [43].

The variable friction category is based on the use of lateral forces that can be used to create the illusion of material features and textures on the display. To achieve these effects, electrostatic concepts [59] and acoustic waves at the fingertips [50] can be used.

Finally, there are the shape changing surfaces, such as a dynamic Braille display from a pin-array type tactile display [60] or the use of piezoelectric contactors to create compression and traction stresses [42].

2.3 Pseudo-Haptic Interaction

The pseudo-haptic interaction consists of simulating some haptic effects, called pseudo-haptic effects, without using a haptic device or actuator, that is, only using graphic changes or visual tricks to provide the effects when tactile interaction occurs [28]. By not using any physical haptic device or actuator, which often cannot be applied, due to the shape or relatively high cost, it is possible to increase the haptic sensation through devices present in people’s daily lives, such as smartphones and tablets, providing a better experience during human-computer interaction.

Pseudo-haptic is haptic information simulated from a sensory conflict when the tactile information differs slightly from the visual information [21]. Sensory conflicts have been studied for decades. An example is the work of Rock and Victor (1964) [46], in which the authors performed an experiment in which they asked participants to touch a cube, looking through a lens that distorted its shape. In this experiment, visual dominance over tactile was evidenced, because the participants when asked about the shape of the cube touched, answered the visualized shape. Ernst and Banks (2002) [16] proposed a model to explain the predominance between human sensory channels. According to the model created, when sensory conflict is related to spatial perception, the human central nervous system prioritizes vision over touch, as it has greater precision. On the other hand, when sensory conflict concerns textures, touch is selected, as it has more precision in estimating the properties of a material than vision. Spatial “relocation” promotes visual dominance instead of visual-haptic integration, considerably increasing the weight of the dominant sense in the perception [12]. The illusions generated because of these sensory conflicts are errors made by the human brain and not by the senses [49]. Goldstein (1999) [19] found that vision can trick a subject during a task of discriminating conformity between two sources that uses touch, showing that vision can be used to generate haptic illusions [49]. Berthoz (1997) [10] suggested that the sensory illusion cannot be considered as a wrong solution or an error, but as the “best possible hypothesis” identified at that moment by the human brain. Hatwell et al. (2003) [21] affirm that the illusions generated from the vision and touch are based on the same rules and representations of real-world properties, that is, the same characteristics identified in previous experiences with real objects are used to identify illusions.

The pseudo-haptic feedback combines visual information in a synchronized way with the user’s movement or motor action during the simulation [28]. In this way, a coherent environment or an environment that becomes coherent can be created for the subject during the interaction, allowing the simulation of effects such as weight, stiffness, relief, texture, friction and viscosity. Lécuyer (2009) [28] makes four main statements about the pseudo-haptic feedback, which are: it implies one or more sensory conflicts between visual and haptic information; it is based on the sensory domain of vision on touch in the perception of spatial properties; it can correspond to a new and coherent representation of the environment resulting from a combination of haptic and visual information; it can create haptic illusions, that is, the perception of a haptic property (characteristic perceived by touching the material) different from that present in reality.

Some limitations can be found in the use of pseudo-haptic feedback, including the use of touchscreens. According to Ujitoko et al. (2015) [55], the problem of occlusion, which is when the finger that touches the screen hides the visual feature presented, is a limitation, as well as decoupling, which consists of changing the speed ranges of the graphic cursors during interaction to identify their locations and the visual illusion.

A relevant effect on haptic sensations is the stiffness of a material. The stiffness is the level of flexion or deformation of a material under pressure or force during a period (time). And in the case of the pseudo-haptic interaction, the level of graphic deformation or distortion of the virtual object touched during a period.

3 Related Work

Several studies have been carried out in the area over the years using the pseudo-haptic interaction to prove perception of effects or measure the level. Most studies used the mouse or some other external device, which can be expensive or unconventional, combined with the pseudo-haptic effects. Besides, most studies did not have a large number of participants [9, 14, 40, 44, 58] which respectively had the participation of 12, 14, 7, 13 and 12 people.

There are works that used physical objects, which can be used in AR systems; or which used motion sensors to assist in interaction. Subsection 3.1 describes works in this category.

There are also studies that are restricted to only making visual changes, known as pure pseudo-haptic (Subsect. 3.2). Few studies used touchscreens and only one used the vibration motor present in devices, such as smartphones.

3.1 Combined Pseudo-haptic

In the literature, there is research to verify if changes in perception occur or improve the user experience when combined with some other device, mostly, that offers physical feedback.

Studies have been carried out that combine pseudo-haptic effects and haptic devices or actuators to provide physical feedback on tactile interaction. There are also works that use a passive haptic device attached to the fingers, combined with visual changes, to generate pseudo-haptic feedback simulating stiffness [2, 23]. Other studies have combined the pseudo-haptic effects with pressure actuators [25, 58] or with a force feedback device [29].

The pseudo-haptic approach was also used in AR environments to simulate mass effects on virtual objects projected on markers [22], as well as in VR environments [4, 26, 36, 48].

Studies were carried out to measure the participants’ perception using the vibration feature. Ridzuan et al. (2012) [45] conducted a study using a vibration feature from a touchscreen device and measured the stiffness effect using this feature and graphical changes to the interface. Hachisu et al. (2011) [20] used a mouse device with vibrotactile feedback in their research. Studies combining pseudo-haptic approaches with a pen can also be cited, with, among other things, the vibration feature [35, 54]. The use of vibration feature is particularly interesting as it can be used in most current smartphones that have an embedded vibration motor.

Studies that modify the reality in the participants’ interaction, hiding their hands or fingers were also conducted, demonstrating that the participants can perceive the illusion created using a pseudo-haptic approach as their reality at that moment in relation to the angle, position, curvature, force and size [5,6,7,8, 27, 44].

Microsoft’s Kinect device, which recognizes body movements to increase interactivity in virtual games, was also combined with a pseudo-haptic approach in one study [18].

3.2 Pseudo-haptic Interaction

The studies classified in the category of pseudo-haptic interaction did not use any device with physical feedback, being composed purely of visual changes. Most of them were conducted using only a computer and interaction using the mouse device. Some studies have used other passive devices that do not provide haptic feedback in experiments [30, 31, 33, 37, 57].

An approach to perform this simulation on devices that do not have a pressure sensor was proposed by Argelaguet et al. (2013) [3]. The authors suggested using time and visual changes regarding the size of graphic objects that represent materials to simulate stiffness. The size changes occurred at the moment of touch, the size of the cursor decreasing to a limit while the finger remained in contact with the screen, and when the release occurred, the cursor returned to its normal size.

Lécuyer et al. (2000) [32] and Lécuyer et al. (2001) [33] worked with the hypothesis that the stiffness effect could be simulated by the visual deformation of an object. For this, they used a Spaceball, an isometric input device with six degrees of freedom, and a virtual object represented on the screen, which had the pressure ratio changed, that is, the more rigid an object was, the less deformation it suffered and the less rigid he was, the more deformation it suffered. Even though Spaceball is not a haptic device, because it is static, passive and does not have pressure reaction feedback, the participants had the perception of different levels of stiffness when interacting with different virtual objects that suffered varied deformations.

Two other pseudo-haptic studies conducted experiments using springs to measure the participants’ perception of different levels of stiffness. In the first study, participants interacted using the Spaceball device and visualized the deformation of the virtual object that represented the spring. In this study, participants answered about similarities in stiffness of a virtual spring in relation to a real spring [32]. In the second study, real torsion springs were used, as well as virtual torsion springs simulated using pseudo-haptic feedback. In this study, participants pushed the springs and had a similar perception of stiffness when comparing real and virtual springs [41].

Works focused on touchscreens were conducted by changing objects in the graphical interfaces for pseudo-haptic interaction. These works verified the perception of the participants in relation to the simulated effects in this type of screen during the interaction [9, 14, 40, 55].

4 Case Study

The main purpose of the case study was to verify and measure the level of perception of the haptic sensation of stiffness when interacting with real and virtual materials, the latter being through pseudo-haptic effects on a smartphone’s touchscreen; as well as checking the pseudo-haptic effect with and without vibration. For this, an experiment was carried out with participants, collecting data on perception through questionnaires. According to Lécuyer et al. (2000) [32] some haptic sensations can be perceived by participants with similar levels when interacting with real objects. It should be noted that the haptic sensation must be caused by visual or graphical changes, deforming the calibrated virtual materials to represent the studied effect of the corresponding real materials, without using any actuator external to the smartphone. Each virtual material was made with the appearance similar to the corresponding real material in a superior view of the material. Real materials were made available to participants to allow comparisons of perceived sensations, according to the literature [32, 41, 58]. The hypothesis is that there were not great differences for the same materials among three scenarios.

Fig. 1.
figure 1

Real materials used in the experiment (from left to right): rubber, sponge, fabric, plastic, cardboard and wood.

Full size image

4.1 Experiment Materials

The experiment had real objects at the disposal of the participants, which consisted of six materials with different levels of stiffness. As can be seen in Fig. 1, following the order from least to most stiff, the materials used were: rubber, sponge, fabric, plastic, cardboard and wood. To represent the rubber a balloon was used. The sponge was represented by the softer side of a common cleaning sponge and the fabric by a common cotton cloth. The plastic was represented by a lid of a common pot, while for the cardboard was used a part from a thick and hard cardboard shipping box. Finally, a piece of solid wood was used, being the only material totally rigid.

The order was defined manually, pressing each real material, since there may be variations of a certain material. For example, there are different plastics, which can have different levels of stiffness. All materials were fixed on individual supports measuring in centimeters (cm), 11 cm \(\times \) 8.5 cm \(\times \) 5 cm, for length, width and height. The materials occupied the same area, had the same shape and were visible only from the top face to the participant.

The experiment also included a smartphone with a native vibration feature to perform interaction with virtual materials through its touchscreen. The smartphone used to carry out the experiment was a Motorola G5 Plus with a 5.5-in. screen, with no pressure measurement sensor on the screen or an external feature. For the creation of the Android application containing the virtual materials, Framework7 v5.4.2 [56] and Cordova v9.0.0 [52] technology were used. To simulate realism, the textures of the virtual materials were obtained from high-resolution images of each corresponding real material. The presentation of the face of the virtual materials was represented by a 4.5 cm\(^2\).

The JavaScript Rebound v0.0.7 library [17] was used to provide the pseudo-haptic effect, allowing the resizing and distortion of an image through tension and friction adjustments. This software library allowed to generate a visual effect of sinking the material (displacement of parts of the texture of the image in direction to the center in a period (time), creating an elastic deformation) when the user touches the virtual material, similar when the user touches the real materials, without changing the borders. The calibration was done manually by the researchers, comparing the visual distortion generated between real and virtual materials, modulating the deformation coefficient for the softness/stiffness of each material setting the friction and tension values.

For the performance of the experiment, a computer lab room with sound insulation, closed door and windows without external view was made available. Inside the room, there was a table measuring \(400\,\mathrm{cm} \times 150\,\mathrm{cm} \times 75\,\mathrm{cm}\) in length, width and height, respectively; and a chair measuring \(55\,\mathrm{cm} \times 57\,\mathrm{cm}\) in length and width, with a seat 42 cm high in relation to the floor.

The camera of an LG K10 smartphone was used to record participant interactions in each task. The camera was focused on the material or smartphone and on the participant’s dominant hand, recording the moment of the interaction. The participants did not have their faces or bodies recorded to avoid their identification. The camera was freely manipulated by the researcher who was conducting the experiment, starting and stopping recording for each task performed. The visualizations of the recordings were used later in the analysis of the results of the experiment, allowing to verify behaviors and also identify if any participant did not perform the task correctly or discrepancies between responses and actions of the participants were found, invalidating the participation.

For statistical analysis, seeking to verify significant differences between the same materials in the scenarios, the R programming language [53] was used. Finally, ballpoint pens and printed questionnaires identified by unique numbers were made available, so that the participants could inform their perceptions during the data collection phase. The numbers would be used to remove data from the survey in case of withdrawal, since the confidentiality of personal information must be respected.

Each questionnaire was divided into three parts, one for each scenario, composed of six questions (one for each material). Only the scenario was described in the questionnaires, without informing the order or descriptions of the materials. At the end of each task in each material in a scenario, the participant was asked to answer the following question: On a scale of 1 to 5, being: 1 - Very soft; 2 - Little soft; 3 - Soft; 4 - Hard; 5 - Very hard, what is your perception of the level of stiffness/hardness when pressing the center of the material? It can be seen that ties were possible, since the scale has five values for the notes of perception and there are six materials.

4.2 Participants and Tasks

The experiment had the participation of nine male students, volunteers, without any remuneration or bonus, from the fifth semester of the undergraduate course in Information Systems, in the age range of 21 to 25 years. As they are from the computing area and from a generation accustomed to using smartphones in their daily lives, no barriers were encountered that could occur with any participant unfamiliar with the device used. Due to the profile explained and the simplicity of the task required to be performed, there was no need for prior training to perform the experiment. The participants only used the real materials and the smartphone of the experiment to perform the task, not being allowed to access or use any other resources during the experiment.

The main task consisted of pressing the center of the material (real or virtual) with the index finger of the dominant hand, so that the participant perceives the level of distortion, as can be seen in Fig. 2. The participants were free to press as many times as they wished, with no time set for execution, in order to have the perception of the stiffness of the material, as long as they always pressed in the center. The material was made available on the table, in front of the participant, who should press it from top to bottom against the table to avoid displacement. In this way, the visual distortion of sinking generated during the experiment, whether in real or virtual material, would be visualized in the same way. No further manipulation with the materials and the smartphone was permitted.

Fig. 2.
figure 2

Participant performing task on materials: (a) real, (b) virtual - before interaction, (c) virtual - during interaction.

Full size image

After performing the task with the material, the participant should answer the question of the printed questionnaire, informing their level of perception about the stiffness of the material. During the task, the objective was to collect the perception in relation to each material and avoid comparisons. Therefore, to prevent the participant from trying to adjust their responses, creating a scale of material levels, the participant could not change a previous answer informed in the questionnaire or perform a task on a previous material again.

4.3 Scenarios

The experiment consisted of three scenarios for each participant, as follows: (i) interaction with real materials; (ii) interaction with virtual materials; and (iii) interaction with virtual materials and with the vibration feature. In both the second and third virtual scenarios, visual distortion occurs, generating a sinking effect when the participant touches the screen of the smartphone. The level of this sinking is based on the stiffness coefficient configured for the material, making the materials behave differently when they are touched. The difference between the second and third scenarios was the use of the smartphone’s native vibration motor, starting when the participant’s finger touched the screen and ending when it moved away from it. The vibration was continuous, with standard amplitude and frequency of the smartphone’s native vibration motor during touch for all materials.

As the experiment aimed at the individual measurement of sensation by each participant, the exchange of messages between participants could influence perception and data collection. For this reason, the participants did not have contact with each other while they were participating in the experiment.

4.4 Experiment Design

Each participant entered the room individually and received an identification number, which was used in the experiment questionnaires answered by him for future analysis. Any question could be asked by the participant before starting the experiment, so that they would feel comfortable and be confident in the experiment. The experiment did not represent any particular type of risk to the participants’ health or required personal or confidential information. It is important to mention that the final objective of the experiment was not informed to the participant, that is, that their responses would be used to measure perception and make comparisons of perception levels. This information was omitted so that the participant was not encouraged to try to place the same level among the same materials in different scenarios.

The participant performed the experiment individually sitting on a chair with the table in front of them. The real materials and the smartphone with the virtual materials were made available individually at different times to the participant, according to the predefined order of the scenarios and materials for carrying out the experimental tasks. The participant only interacted with the next material after indicating the perceived level of stiffness of the material in the scenario questionnaire and only advanced to the next scenario after interacting with all the materials in the scenario.

The experiment had a researcher instructor in the room to monitor the progress. He informed the participant of the names of the materials used in the experiment. In the smartphone screen was presented the name of the material at each moment of the experiment. The instructor’s role consisted only of: (i) conducting the sequence of the experiment observing the predefined sequence; (ii) providing basic instructions and ethical information without interfering with the participant’s perception; (iii) delivering and collecting the real materials, the smartphone with the application and the questionnaires that were used in each task; and (iv) filming the participant’s hand performing the proposed task for further verification.

The order of the scenarios varied by participant, using the possible combinations between the three scenarios. Thus, one participant started the experiment by manipulating real materials while another participant manipulated virtual materials without vibration, and a third one manipulated virtual materials with vibration. Likewise, the possible combinations related to the real and virtual materials available in each scenario were used. Thus, one participant started with Wood material, while another with Sponge and a third with Rubber in the same scenario. Using the combinations between the scenarios and between the materials, each participant performed the tasks following a different order in the experiment.

The participant was not allowed to: (i) return to a previous scenario; (ii) have access to the materials previously, except at the time of the task; and (iii) have access to the order of availability of materials. The researcher instructor was the one who individually delivered and collected the material at the time of the task, following the predefined sequence without informing the subsequent material or scenario.

5 Results

All participants performed the three scenarios individually, interacting with all the materials available for each scenario. In total, 162 responses were collected on the participants’ level of perception. The average levels by material and scenario, as well as their distribution, can be seen in Table 1 and through Fig. 3a. The following subsections detail the results considering the goals of this paper, to: (i) prove the pseudo-haptic effect perception in smartphones for certain materials (Subsect. 5.1); (ii) measure the levels of perception (Subsect. 5.1); and (iii) check the sensations with and without the use of vibration (Subsect. 5.2).

Table 1. Overall average of the participants’ perception level according to the scale: 1 - Very soft; 2 - Little soft; 3 - Soft; 4 - Hard; 5 - Very hard.
Full size table
Fig. 3.
figure 3

(a) Perception levels of stiffness (Y axis) among materials (X axis) (b) Perception levels of stiffness (Y axis) among real and virtual materials (X axis).

Full size image

5.1 Perceived Level of Stiffness

Analyzing the averages, it can be identified that the participants perceived the pseudo-haptic effect of stiffness in virtual materials, with and without the use of vibration, as there is a gradual perception of the stiffness level in virtual materials according to real materials (Fig. 3a). It is important to highlight that some similar responses would be possible, especially between pairs of materials, since there were six materials and five levels on the scale of responses. Thus, the perception of the pseudo-haptic effect of stiffness by the participants was proven. Additionally, the level of stiffness was measured, following the proposed scale, although it was expected that for the materials of the extremes (totally soft and totally rigid) the responses would not present discrepancies.

However, when comparing the averages of the participants’ responses for virtual materials without vibration with the real materials, it can be seen that in most comparisons between materials, the virtual material was perceived to be a little stiffer than the corresponding real material. The results are shown in the scatter plot of Fig. 3b.

The Friedman nonparametric test was applied to identify statistically significant differences between the three scenarios for each material, checking the hypothesis. The reason for using this statistical test is related to the data and analysis characteristics, such as: (i) the residuals of the data does not have normal distribution; (ii) there are more than two sample groups for comparing; and (iii) the same individual provides more than one response of perception, participating in different scenarios.

After applying Friedman test, no significant differences were found (hypothesis corroborated), since a \(p-value \le 0.05\) (confidence level at 95%) was expected, and the \(p-values\) were higher than 0.05 for all materials. For the Rubber material, it was obtained a Chi-Squared or \(X^2 = 0.4\) and a \(p-value = 0.8187\); the Sponge material reached a \(X^2 = 2.7692\) and a \(p-value = 0.2504\); for the Fabric material the test presented a \(X^2 = 2.1739\) and a \(p-value = 0.3372\); and for the Plastic material the result was a \(X^2 = 0.63636\) and a \(p-value = 0.7275\). For the Cardboard and Wood materials the \(X^2\) were 1.3333 and 2.0; and \(p-values\) were 0.5134 and 0.3679, respectively.

In the end of some sessions of the experiment, three participants made comments about the study. Two of them said that they needed to add force during interaction with virtual materials that had higher stiffness levels; and one of them commented that it was not necessary to add force to press the materials because the screen of the smartphone is not flexible.

5.2 Combination with Vibration Feature

When the analysis is related to the averages of virtual materials, comparing the use or not of the vibration feature of the smartphone, the result was that the participants, in most materials, perceived a lower level of stiffness with the feature than without it, as shown in Fig. 4a. Except for Rubber and Cardboard, all other materials had a reduction in the perceived average level. In the case of the Cardboard material the average was equal with and without using the vibration feature.

Fig. 4.
figure 4

(a) Perception levels of stiffness (Y axis) among virtual materials with and without vibration feature (X axis) (b) Perception levels of stiffness (Y axis) considering each pair of real and virtual materials with vibration (X axis).

Full size image

6 Discussions

From the results it is possible to evidence that the simulation of the stiffness effect using a pseudo-haptic approach was perceived by the participants, and that in most materials the level of perception was close to the level of the real material. These results were obtained without previous training with the participants, restricting the participants’ access to the materials only at the moment of the task. The names of the materials were not identified in the questionnaires, it was not allowed to change responses attributed and different combinations were used for presenting to the participants the three scenarios and the six materials. Thus, each participant carried out the experiment following a different order, which did not respect the scale of stiffness of the materials and which could start with a scenario containing virtual or real objects.

Discrepancies on the extreme materials (totally soft and totally rigid) were not expected. Additionally, the tie between the materials on the scale was expected, especially between two approximate materials at the stiffness level, since there were six materials and five levels on the scale.

The Sponge material had the higher difference in relation to the scenarios according to the statistical test, although the difference is not significant. This occurred possibly due to the visual presentation, which although based on the image of a real sponge, the predominantly yellow color did not contribute to the representation of this virtual material, making it difficult to visualize when the participant performed the task, because the stiffness effect is caused by a visual distortion which causes the image to sink when touching according to a predetermined setup.

On the other hand, Rubber and Plastic, that had an appearance considered rich, obtained the values closer to each other for the three scenarios. This can be seen in Table 2, which shows the standard deviation between the scenarios.

Table 2. Standard deviation between three scenarios for each material.
Full size table

Variations in the level perceived by participants was expected in this study, even with the hypothesis corroborated, because the stiffness perception could vary according to previous individual experiences with the materials. The case of the Wood material can be used to exemplify this perception difference, since it was the only material in all scenarios in which there was no distortion or deformation in the touch, and four responses from participants (two after handling real materials and two after handling virtual materials with vibration) were not the level 5, the maximum scale stiffness (fully stiff material). The use of other resources, such as sound effects, can help, reducing the doubts in the perception.

Analyzing the responses from participants, it can be seen that most of the responses varied little with respect to the calculated average level of the participants for each material, except for two cases in the scenario of virtual materials with vibration, which did not follow the presented pattern and can be seen in Figs. 5a and 5b. The participant 8 identified the Rubber as a stiff material, defining the level 5 (highest level of stiffness), and the participant 5 identified the Wood as a soft material (level 2). If these responses were disregarded, the average in the virtual scenario with vibration for the material Wood would be 4.7, compared to the averages 4.8 and 5.0 in the other scenarios for the same material; and the Rubber material would have an average lower than 1.4, similar to this material in other scenarios, with a value of 1.2, also reducing the perceived level when using the vibration feature as in most other virtual materials. It is important to highlight that these conflicting values were not the responses to the first task performed by the participants in the experiment, and in the other scenarios these participants answered according to the average for the material. It is also important to note that the recorded videos of the tasks of all participants were evaluated and problems were not identified, such as doubts and unexpected movements during interaction.

Also when analyzing the videos of the recordings made at the time of handling the virtual materials, it was found that most participants seemed to apply a certain amount of force with their finger to press depending on the material, trying to identify the level of stiffness. This analysis shows that even the participants being from the computing area, with knowledge about the device used in the experiment, that is, knowing that the screen was not flexible and did not have any type of pressure sensor, they needed to apply a certain force to try to complete the task of pressing the materials.

The information provided about names and textures of the materials could have influenced the responses. The participants could think about levels of stiffness according to previous experiences with these materials. However, each material has variations, e.g., a plastic can be hard or soft, depending on its composition. This could cause problems in the perception because of previous experiences with these materials. We observed that the participants strove to identify the stiffness of each material based on visual distortions provided in that moment.

Fig. 5.
figure 5

Responses for the scenario of virtual materials with vibration (a) Rubber and (b) Wood.

Full size image

The use of the vibration feature of the smartphone in virtual materials, provided the participants the perception that most materials were less stiff, and in some cases even less than some real materials, such as Plastic and Wood, which can be seen in Fig. 4b. The results showed that the vibration can mitigate the differences perceived, since the screen of the smartphone is not a flexible surface, such as a real material that is not stiff, and the deformation does not happen during the interaction. Except for Plastic and Wood, the responses to other real materials presented average values of perception less than the mean values for the same virtual materials when the vibration feature was employed. This observation evidenced that the addition of the vibration must be rigorously studied, because it can influence the pseudo-haptic effect perception.

Finally, the number of participants of the experiment is similar to the numbers found in the main related work. However, we believe that it is important to carry out the experiment with a higher number of participants, including different profiles, to compare with the results obtained in this study.

7 Conclusion

The current study aimed at proving and measuring the stiffness effect perception using a pseudo-haptic approach for certain materials, as well as checking the use of the vibration feature, through a smartphone with touch screen. The haptic feedback can improve the users’ experience during the interaction with systems.

According to the results of the experiment, the participants perceived the effect, and the stiffness levels perceived were similar for each pair of real and virtual materials (considering the same material type). When comparing the perceived level between real and virtual materials, a small increase, without statistically significant difference between the scenarios, in the level of stiffness in the perception of the participants was observed for most virtual materials compared to the real ones. A comparison was also made between the virtual material scenarios without the vibration feature with those that had this feature. The results showed that the participants perceived a lower level of stiffness when the pseudo-haptic effect is combined with the vibration feature for most materials than when this effect is not combined with the same feature.

The contribution was to evidence that is possible to develop computational systems, especially systems based on VR and AR, with pseudo-haptic interaction using graphical interfaces of devices with touchscreens and vibration motor, at least for the profile of the participants of the experiment. These interfaces, during the human-computer interaction, can stimulate two sensory channels (vision and touch), improving the immersion and the experience of the user to perform tasks.

For future work, other materials can be used to measure the perception of the pseudo-haptic effect by participants. Materials with different appearances, formats and sizes could be used. Other tasks can be planned, such as: moving or grabbing the materials. Experiments in which the participants do not receive information, such as the name and the texture of the materials, could be made. As the human brain uses several sensory channels scattered throughout the body to identify properties of real materials, the addition of a sound effect in the interaction would activate another channel and, with vision and touch, could assist in the perception of stiffness.