1 Introduction

Human–computer interaction HCI is an interdisciplinary discipline addressing the intersection of people and technology [24], which has evolved strongly from the first computer programs where all events were based on textual input–output devices [71]. As the interactivity of the software grows, the importance of HCI research intensifies and new advances appear to improve the relation human–computer. A lot of HCI advances have emerged from the knowledge of the human mind, which can be visualized as a processor divided into three interacting subsystems: (a) a perceptual system (managing visual and auditory stores generated from the stimulus), (b) a cognitive system (making decisions about how to respond the stimulus using long-term and short-term memories) and (c) a motor system (responsible for carrying out the response) [6]. In this line, HCI research is being made on critical human issues, such as perception, cognition, emotion and multimodal interaction [24].

From a developmental and planning perspective, software, at least highly interactive software, must be conceived and developed with HCI as one of its higher priorities. In order to ensure good HCI design, a number of principles must be respected. Following [32] these principles should include: (a) knowing the user (interaction and interface must fit the needs of the target user), (b) understanding the task (users must comprehend the job to be completed through the use of an interactive system), (c) reducing memory load and refresh the user memory, (d) striving for consistency, (e) preventing errors and allowing an easy reversal of the erroneous actions and (f) reflecting a correspondence between the actions in the system and the actions in the daily life.

With varying degrees of wisdom and balance, existing software systems implement very different types of interaction. According to [57], this interactivity can be classified depending on the mechanism used to interact: (a) standard interaction through the keyboard, the mouse or other remote control allowing point & click, (b) interaction using special peripherals such as a steering wheel, a joystick or a guitar (for instance), (c) tactile interaction, touching the screen with a single finger movement to select the available options, (d) interaction by voice, (e) active interaction, where the user employs different peripherals (e.g., gloves, video cameras) or performs gestures to interact with their own body while devices as gyroscopic sensors or accelerometers are used to capture their movements and (e) pervasive interaction, which integrate the context of the user as part of the system and objects in the environment become also interactive. Laws like Fitts [14] and its variants [42] have been fundamental to improve the design and construction of interfaces, guiding in addition, efforts to achieve simple and friendly interfaces. A consequence of these studies are the advances that have experienced the natural user interfaces (NUI), which try to integrate hardware and software to achieve a much more direct and efficient interaction between the user and the computer [15].

In this point, NUI emphasize the HCI principle of naturalness and integrate several of the types of interaction mentioned above. Thus, the underlining desire in this field, that is, the invisibility of the human computer interface, requires the capacity to maintain a conversation in natural language or to interpret human gestures together with the adoption of a metaphoric model in the system. Inevitably, this interactive structure requires a process of learning by the users who must know the range of verbal vocabulary and/or body gestures allowed by the system. However, this initial effort enables unique opportunities during the interaction [13] and natural interfaces are continually incorporated to more and more systems in different domains, such as interaction in virtual reality environments (Lv et al. [41]), interaction with robots (iCub applied by Li et al. [40]), mobile healthcare (m-health) (OTELO system applied by Garawi et al. [18]), learning contexts (WikiNect development by Mehler et al. [43]), just to mention some examples.

NUI allow users to be able to feel completely immersed into the devices they use [58], and immersion is precisely the main characteristic that promotes more effective user-system communication in NUI in comparison with the traditional graphical user interfaces (GUIs) where the mode of interaction is supported by WIMP stands (windows, icons, menus and pointers). Within WIMP, the touch screen user interface (TUI) is a special case of GUI to achieve interaction. The aim behind the natural paradigm is to remove barriers and offer a style of interaction more similar to the utilized in the real world; conveniently, the use of gestures can provide an attractive and natural alternative [31]. Following these authors, gestures are a body movement (through fingers, arms, hands, head or the face) which transmit meaningful information or are used to interact. Gestures can be static (a fixed posture) or dynamic (a continuous change of the posture with time) and it is necessary to specify the beginning and end of each gesture in terms of time and space [31]. Some gestures may have different meanings depending on the context (previous and subsequent gestures). In consequence, gesture recognition is a non-trivial task which must be tackled with meticulousness to achieve an expressive communication with the user.

Besides, and not less important, this new form of interaction enables users to distance themselves from the devices being handled, allowing the system to take into account the diversity of its users [16]. Consequently, individuals with certain disorders such as Parkinson’s disease, dyspraxia, motor neuron diseases and paralysis could benefit from gesture interaction. Similarly, people with temporary problems in their motor skills and seniors with difficulties during the performance of tasks requiring fine motor ability could find a very acceptable solution in this new paradigm of interaction. Particularly, the same applies to those confined to bed or who must remain in a position that prevents the habitual use of an electronic device. People with severe visual difficulties could also overcome many persistent obstacles they suffer from their daily relation with software applications (especially in mobile devices). Likewise, gesture-based interaction may be embraced as an additional or complementary mode of interaction in a multimodal system (in line with the multimodal interaction that regulates life itself [70]). This allows any user to benefit from this alternative, introducing natural interaction by gestures as an instrument to achieve a universal access.

On the other hand, the user’s need in communication is prevailing and today it is the smartphone that covers this need, presenting the user a TUI to achieve the interaction. The industry tries to provide new and better features to these devices, in terms of both hardware and software, becoming a powerful computer system, despite its small size [67]. However, this rapid development, in many cases, has ignored special needs (momentary or definitive) of the user [30], for example, having to adopt special anatomical positions of body typical for sick or rest (lying down). By disease, the dorsal decubitus position (the person lying on his/her back, parallel to the ground) is the most common and also possibly the one that causes greater discomfort and imprecision in the use of TUIs, which in other circumstances would be easy and accurate. The problem for the user is complicated ever more when restrictions of movement in their extremities are added (arms, legs, neck) or even vision (e.g., eye surgery), which limits the use of resources of these devices.

This paper analyzes the potential advantages that a natural user interface may have by contactless gesture (GBUI) versus a TUI, applied in the handling of a mobile device at different anatomical positions. The more common anatomical positions that hinder the handling of the device and affect the accuracy of an action on a mobile device (henceforth considered in this study) are standing and lying down. These are the two more usual positions in everyday life, for healthy people the first, and for sick, disabled or comfortable people, the second. Subsequently, and taking into consideration the positive results of the experimental study performed by 25 users, a proposal of interaction supporting GBUI to make phone calls with smartphones has been made. This study case has been chosen for being the most common and obvious use of this devices. The proposal includes an architecture specific for this type of interaction and a software solution based on it. The software solution is independent of the anatomical position of the user, allows to manage phone calls, operates with a proper phone book, proves a voice help and allows to personalize the gestures implemented in the library.

The paper is organized as follows: Sect. 2 offers a panoramic view of the related works that use gesture interaction and raise its application in several areas; Sect. 3 presents the proposal and the research approach; Sect. 4 shows the design of the iteration tests and the evaluation parameters. Section 5 outlines the software tools used to apply the interaction test; Sect. 6 explains the testing TUI versus GBUI, experimental protocol used in trials, and discusses the obtained results; Sect. 7 presents a specific GBUI application executed on mobile devices as a case study, determining how it is designed and validated; finally, Sect. 8 presents conclusions and further works.

2 Related works

We present previous works related to the use of gesture in HCI according to several aspects: gesture interaction, gesture recognition, gesture application and gesture accessibility. Table 1 summarizes the analyzed works.

Table 1 Overview of studies using gestures including our proposal
Full size table

2.1 Gesture interaction

Multimodal human–computer interaction (MMHCI) poses new paradigms and establishes a communication model, where the context of the interaction has a significant impact on aspects such as the information that is transferred, when the transfer was made and how it was performed [69]. This requires understanding three aspects: the system (the computer technology and its usability), the user who interacts with it and the interaction between the user and the system [27]. In this context, MMHCI seeks to combine several techniques to build a different type of natural interface where the user interacts through their senses [27]. However, even though the human being has five senses, so far no form of interaction manages to combine all of them. Fortunately, the gesture offers a great advantage in the interaction, because it combines sight (gesture image), touch and knowledge (meaning) [48].

The application of gesture in communication with the machine is not new, and many research works have been oriented to achieve interaction using body [17, 66], facial [75] and manual gestures [11, 12, 22, 33, 50]. In fact, the gesture in the human being can be considered a form of communication [46]. In this sense, the gesture is a typical feature of our species that best adapts to achieve a natural interaction with the machine [61]. The hand gestures, in relation to others generated by the human body, may be so simple that they go unnoticed, or so complicated that it even causes pain in performing them [62]. The gesture may be performed on surface (contact) or on air (without contact). In addition, depending on whether the gesture involves movement or no, it can be classified as dynamic or static, respectively [46]. The capacity of expression of the manual gesture is such that it ranges from simple libraries (one or several gestures) to complete gestural languages called sign language. The most used sign language is the American sign language (ASL) and its variants for different languages and regions.

2.2 Gesture recognition

Various methods have been used to capture and interpret the gesture, some of them combining hardware and software [23], using gloves [77], images [1], infrared devices [12] and sensors of multiple and varied types [12, 33, 54]. These methods have been driven largely by the significant development experienced in signal processing technologies and mathematical models, admitting in its refinement techniques of artificial intelligence (AI) [3, 8, 55, 59]. All this has allowed to identify and classify the gesture more quickly and effectively [2, 8, 36, 49, 55]. Within these methods, the recognition of hand gestures based on electromyographic (EMG) signals has obtained very good results [8, 55, 74]. EMG signals present in the muscles activity (arm and forearm) permit the recognition of the hand gesture. The EMG signals are detected by electrodes, which may be of surface (sEMG) or intramuscular (iEMG) [21], whose measurement ranges from microvolts to millivolts. These ranges are directly dependent on the signal strength, in addition to location and type of electrode [3, 25]. The results in the identification of this type of signal using sEMG electrodes have proven to be highly effective [53]. For example, using a linear array of electrode [74], a sEMG wireless set [8] or a set’s sEMG placed around the arm [39]. In this context, Al-Timemy et al. [3] achieve a correct identification of gesture movement in people with amputated limbs. Currently, wearable sensors incorporate EMG electrodes and inertial measurement units (IMUs) that capture the biosignals [35, 53] and motion data, and transmit these information by Bluetooth or WiFi [7, 55].

2.3 Gesture application

Along with technological advancement and in synchrony with the search for new and better interfaces, the gesture has been applied in multiple contexts, ranging from the execution of certain actions (e.g., movements in games) to complete solutions of management and control (e.g., management of home devices) [54, 56]. Examples include control devices [20, 44, 54], limb replacement prosthesis [3, 59, 60] and medical diagnosis [5, 19].

Several research works have been presented in which the smartphone is used in combination with gestures. These works use the resources offered by mobile computing devices and exploit their computationally power and rich functionality. Among all its functions (agenda, notes, photos, videos, etc.), communication between people (messages, calls, etc.) is perhaps the most relevant, as was determined by Jung et al. [29]. However, also, the smartphone has been used in others areas, such as remote control of household devices [34], point of sale in e-commerce applications [4], monitoring device on u-health platforms [45], navigation assistant and geographical location help [9, 10] and tour guide [73]. All these solutions interact with the user through the contact gesture (“touch”) [29], of one or more fingers, on a surface or touch screen (resistive, capacitive, etc.) within a specific area [28, 51]. The touch can be performed in static or dynamic form. For example, the mentioned tour guide [73] uses dynamic touch (zoom gesture) to expand map areas, while in the case of control of household devices presented in [34] the static touch is used to achieve the remote control. With respect to the use of one or two hands to apply the gesture, Lee et al. [37] determine that in most cases the user utilizes only one hand, which causes fatigue, sometimes pain, and increases stress. In addition, the touch screen enables the user to interact directly with the information that is displayed, though it requires accuracy in touch gesture (static and dynamic) [26, 51].

2.4 Gesture accessibility

The degree of comfort and accuracy of the user has an important role in the design of gestural interfaces [62], and its application has been remarkable both in home [54, 55] and industrial applications [50]. However, the affectation to the senses (sight) can make a simple gesture, such as touch, difficult to realize with accuracy [28, 38]. Likewise, the motor skills in humans may be affected, either by accidental causes or as a result of a degenerative problem (disease) in the central nervous system [26, 72]. Similarly, a loss of accuracy in the movement and coordination occurs due to a person’s age or his/her anatomical position at some point [28, 38]. All these aspects compromise, temporarily or permanently, fine motor skills of the hands of a person [16, 52, 64], causing errors in handling devices that require some degree of accuracy.

To help overcome these problems, many accessibility solutions have been proposed (e.g., voice recognition, free hands, image recognition), some of them embodied in devices and operating systems (OS). An operating system that has incorporated accessibility solutions (e.g., keyboard zoom to facilitate the touch) is Android. Nevertheless, these accessibility solutions have not had a significant impact on the degree of user acceptance, especially in older adults [76]. Thus, incorporating a hand gesture interaction option to smartphone management would be a good alternative. In fact, those people who at any given moment (illness or accident) suffer problems of mobility, loss of vision, deafness, etc., would have greater opportunities for communication and use of a smartphone. For healthy persons, the hand gesture interaction may be useful to resolve certain circumstances (screen break, busy hands, special position, etc.), which is important because today smartphones have become a universal communication medium [51].

2.5 Importance of this research

As is shown in Table 1, several reviewed works use the camera for gestural recognition. However, these solutions must handle problems of occlusion, illumination and focus. For its part, other proposals use touch gestures through a touch screen, forcing the user to have accuracy in the touch and good visual recognition. Finally, those using sEMG electrodes offer a better guarantee of gesture recognition at the cost of being, little or moderately, intrusive. Hence, smartphone applications based on sEMG electrodes that implement wireless communication could become an integral system of support to the user. Solutions of this type should be integrated within the OS provided by the manufacturer, and would greatly overcome the limitations of using the touch screen on smartphone devices.

In conclusion, a solution based on gestures without contact, using a personalized gestural library and adapted to the mobile device, that allows the management of applications or services in a smartphone, would be very helpful. During the performed revision, no previous work proposing smartphone applications which use EMG signals for gesture recognition has been found. Additionally, previous studies that analyze the index of difficulty and accuracy in this context have not been found either. However, this is essential to guaranty the useful of the gesture. Consequently, the present paper attempts to address both detected shortcomings.

3 Approach

Our aim is to assess the suitability of gesture-based user interfaces in mobile devices. To achieve this, it is necessary to perform a study that allows comparing this new interface with respect to a classic touch user interface. Thus, we present an analysis based on the difficulty and loss of accuracy of the user when interacting with the mobile device using TUI and GBUI, alternating between two different anatomical positions. These positions are “normal” when the individual is standing and “special” when the individual is lying down. Consequently, this paper maintains the following approach:

  • Design of the interaction test (Sect. 4) for the subsequent comparison between TUI and GBUI. In order to define the test, a previous analysis of a set of smartphone GUIs was performed. This study identified the so-called Importance Zones on which the iteration test is defined. The evaluation parameters of the interfaces are also established to comparison;

  • Development of software applications for testing the difficulty and accuracy (Sect. 5), one for each type of interface (TUI and GBUI);

  • Testing TUI versus GBUI (Sect. 6), where a total of 25 volunteers participated in a total of 250 trials. In the GBUI, the gesture recognition is achieved using a wearable that captures the electromyographic signals through sEMG electrodes. The wearable is placed on the user’s forearm and sends data to a mobile device using Bluetooth protocol. This allows freedom of movement of the hand and greater comfort to the user. Additionally, the concept of gesture control tree (GCT) proposed by [54] has been implemented to determine its validity in the gesture navigation and the degree of acceptance on the part of the user. An analysis of the difficulty and accuracy of the two types of interface, per position, was also performed and discussed;

  • Definition of an architecture for GBUI. This architecture is based on layers on which the application called SICLLE (emergency call control interactive system) has been implemented (Sect. 7). SICLLE is compatible with Android devices, and the gestural navigation is achieved by applying the concept of GCT. The gestures are processed and identified turning into commands for phone calls and contact book management. The system has a library of voice tags to guide the user; in this way replacing visual monitoring. In addition, it is possible to customize the gestures defined in the system associated with each action or command, making it even more user friendly. A brief usability study of SICLLE was performed to validate user acceptance in which participated all volunteers of the test of difficulty and accuracy.

4 Design of the interaction test

4.1 Analysis of devices and interfaces

Smartphones, as obvious, cover a large segment of the mobile market geared to make “telephone calls.” For this reason, the interaction tests were designed for this particular case, although it may be argued that results obtained could be extrapolated to any mobile device (e.g., tablets). Other types of tasks in a smartphone could be useful for analysis, such as handling the phone book, opening and closing applications and accessing configuration options.

The user interaction in the last generation of smartphones is managed currently by the use of a touch user interface and not by the use of a natural interface based on gestures. The screen is built by a multi-touch display, which allows the pressure of one or more fingers. The interaction is perceived when a finger presses a specific area of the screen. In general, this area is graphically represented by a widget or a button in order to ease its identification by the user and the consequent action that the application should perform next.

To design the tests, we first carried out an analysis aimed at determining the average characteristics that the smartphone to be used in the tests should meet, and a study of the user interface of the applications installed by different manufacturers on smartphones to make phone calls. In both cases, 12 smartphones from different manufacturers (Samsung galaxy S6, Sony Xperia Z5, Motorola Moto E, Bq Acuarius A4.5, Sony Xperia M2 and others), grouped into three categories (high, medium and low range), were analyzed.

In the first analysis, it was possible to determine that a midrange device with screen size between 4 and 5 inch, weight between 130 to 155 g and a resolution of \(960 \times 540\) pixels, would be more appropriate to carry out the tests. Consequently, we have used a Sony Xperia M2 smartphone with screen size of 4.8 inch, weight of 148 g, resolution of \(960 \times 540\) pixels and dimensions of \(139.9 \times 71.1 \times 9.6\) mm for testing.

In the second study, it was possible to determine that the user interfaces in charge of call management on different smartphones manifest a similar structure with common areas, in spite of each smartphone having a specific customization layer or a specific Android version (Lollipop and Marshmallow). On the other hand, we determined that the user follows an equivalent navigation sequence in the interaction with the applications. Nevertheless, no custom or accessibility options were activated.

4.2 Importance zones

Common areas define specific zones that the application can identify to manage the phone calls. We named Importance Zones these areas, which are shown in Fig. 1 and described below.

Fig. 1
figure 1

Importance Zones on a Android smartphone

Full size image

Recent calls zone This zone allows the user to select and navigate among the recent calls. Calling from this zone requires to slide a finger from right to left or top down, identify the contact and finally touch the selected contact.

Contacts zone This zone allows the user to scroll through their contacts and choose the one he/she want to call by phone. It requires the same steps as the previous zone.

Information zone It allows the user to see the contact which is being called. It is also displayed when an incoming call is accepted. It requires a visual control.

Decision zone In this zone, we can accept or end a phone call. For accepting a phone call, it is necessary to touch the zone and keep pushing down, while the finger is sliding from left to right, make a move of touch and push down and slide the finger either right, left or up. To end a phone call, a touch to this area is necessary.

For a TUI, the natural way to interact with Importance Zones is by touching consecutively these zones in an order following a specific sequence. Whenever an interaction by a touch is perceived by the system, the system reacts executing an action as long as the interaction is occurred within the importance zone.

For a GBUI, the interaction is made in a different way because there is no contact with the device. The system requires a mechanism to recognize and identify the gesture that user is making. In this study, we selected a gesture-based user interface based on the contraction and extension of the forearm muscles by the analysis of the EMG signals captured by sEMG electrodes of an armband. The interaction is perceived whenever the system is able to identify a specific gesture from a gesture catalog stored by the system. Once an interaction is determined from the identification of a gesture, the system reacts executing an action.

4.3 Definition of the interaction test

The interaction test has been defined on several sequences over Importance Zones identified in Fig. 2. We defined Interaction Point to any instant of time on which the system perceives an interaction. In a TUI, the interaction point takes place when a user touches an Importance Zone. In contrast, an interaction point in GBUI takes place when the system detects the making of a hand gesture.

Fig. 2
figure 2

Sequences of interaction points defined over Importance Zones on TUI

Full size image

The interaction test is composed of three stages. Each stage comprises a sequence of interaction points that must be accomplished following an order. The satisfaction of each interaction point is a goal to be achieved. The goals can be represented by nodes, graphically. Then, touch-goals and gesture-goals are the two kinds of goals that the user must satisfy with respect to each UI.

Figure 2 shows an example of the correlation between goals and Importance Zones which contains the goals with respect to the TUI. Thus, we have: (a) Stage 1, in which all goals are fulfilled (brown continuous line); (b) Stage 2, in which 1, 2, 3 and 7 goals are achieved (green dashed line); and (c) Stage 3, in which a subset of 4, 5, 6 and 7 goals are achieved (blue dotted line).

4.4 Definition of evaluation parameters

The parameters to be evaluated in the interaction test are the interface difficulty index and the accuracy of the interaction. The interface difficulty index is only applicable to the TUI, while the accuracy of the interaction is applicable to both interfaces.

Index of difficulty. Taking into account that it is necessary to execute fast and precise movements which have their limitations on the user’s ability to execute them, we followed the Law of Fitts to optimize the design space [14]. According to the Law of Fitts, the Index of Difficulty (ID) for a motor task [42] is expressed as a relationship between twice the amplitude or distance A and the width W of the ultimate goal as quickly and accurately as possible (accurate, for the moment, implies a small but consistent error rate), in terms of base-two logarithm. The ID is expressed as follows:

$$\begin{aligned} {{ID}}=\log _{2}\left( \frac{2A}{W} \right) \end{aligned}$$
(1)

Because A and W are both measures of distance, the ratio within the logarithm is without units [42]. Equation (1) evaluates ID by attempting to reach the necessary steps to make a call, at any position, from each of the importance zones described in Sect. 4.

Accuracy of the interaction. To measure the accuracy (ACC), we used the definition proposed in previous works [54] that expresses the ACC as a relationship between the successes (positive) divided for all made attempts (positive more negative) to reach a goal. This relationship is described in Eq. (2).

$$\begin{aligned} {\text {ACC}}=\frac{{\text {Positive}}}{{\text {Positive}}+{\text {Negative}}} \end{aligned}$$
(2)

5 Software applications for testing

Two applications were designed to execute the interaction tests, one for each type of interface (TUI and GBUI). Both applications were developed for Android devices. In each of them, the user follows a sequence of interaction points to achieve a goal, in our case related to the management of callings.

Applications for TUI. The touch is a user-defined gesture for touch-based interaction which requires a direct contact of the finger with the touch-surface (touch screen). The interaction points are activated whenever a touch is made within a specific Importance Zone, as shown in Fig. 3.

Fig. 3
figure 3

Distribution of touch-goals (icons) for the TUI interaction test

Full size image

Furthermore, the distribution of the Importance Zones is essential to achieve an optimal performance of the interactions between the user and the mobile device. For the design of the interaction test, Google’s recommendations for graphic interface design on mobile devices were taken into consideration. The separation and size of icons (11.5 mm) tried to fit the study carried out by Jin et al. [28] who verified that larger sizes do not produce a significant loss of reaction time (RT) of the user and therefore do not affect their performance on TUIs. Each touch-goal is numbered and can have two states: active or inactive. In the active state, the goal is visible and is waiting to register an action. When the status is inactive, the goal is hidden and no action can be registered on it. Figure 3 also shows the distribution of nodes through the screen in the interaction tests.

Applications for GBUI. In this case, the user can make gestures without contacting any surface of the smartphone, whenever the gesture was previously stored in a gesture catalog. The identification of the gesture is the significant element which activates an interaction point in the mobile device. It is achieved in our case by a recognition process from the EMG signals collected by the sensors of the mobile device. For this reason, every gesture has associated an interaction point (and consequently a gesture-goal) with the corresponding action when the goal is accomplished. The predefined gestures in this study are described in Table 2.

Table 2 Gestures used in the gestural interface
Full size table

Unlike TUI, the gesture-goals can take place at any time whenever a valid gesture is identified; no important zones should be determined before activating the interaction points. However, the system must control that interaction points occur following a specific sequence to be considered valid as in TUI, as shown in Fig. 4.

To facilitate the tracking of the gesture-goals sequence in GBUI, the gestural control tree (GCT) was implemented [54]. GCT arranges the identification of gestures in separate levels in order to know what the next valid gesture is from the previous valid gesture within the sequence. That is, when a gesture is recognized in a specific level, the system can only identify a gesture of the next level. In this way, the gesture cannot take place at the same time because GCT levels impose restrictions in the sequence. The GCT used for the GBUI interaction test is shown in Fig. 4.

Fig. 4
figure 4

Sequence of gesture-goals for each stage in GBUI interaction test

Full size image

6 Testing TUI versus GBUI

6.1 Experimental protocol

Twenty-five volunteers participated in the experiments to compare the TUI interaction with the GBUI interaction, using everything designed in Sects. 4 and 5. All volunteers had experience in computer science and strong knowledge in the handling of mobile devices. The ages ranged from 20 to 50 years (\({\text {Mean}}=33.44\), \({\text {SD}}=8.7\)) (six females and 19 males). All were owners of a smartphone and had experience with “hands-free” utilities.

The positions used in this study were normal and special. In the normal position, the user was standing with both feet resting on the floor. In the special position, the volunteer was lying on a fully reclined sofa, keeping his/her head without any support (pillow), with a direct view of the screen of the smartphone at the level of his/her eyes.

A total of five trials per participant were performed for each position (normal and special). Each trial had a duration of 10 s controlled by software. During this time, the user was executing the interaction test and the system was registering every attempt made by the user. An attempt is considered valid when a valid interaction point is accomplished within a sequence; in all other cases, it is annotated as invalid.

To calculate the ID and the ACC in TUI, all the touch sequences defined (Fig. 5) have been used. For the calculation of ACC in GBUI, the successes (also called within-goal or positives) and errors (also called without-goal or negatives) were considered.

6.2 Testing TUI

In this experiment, the users executed the interaction test using the software application implemented for TUI. The use of the hands during the trials was the following: A hand was used to hold the mobile device and the other hand (dominant hand or hand of gesture) to perform the touch. It was not allowed to change of hands once the trial was started.

The trial started when the user touched goal 1 of the first stage and then continued to touch the next goals of the sequence, as shown in Fig. 5. The sequences of the interaction test are described below:

  • Stage 1: The volunteer touched goals 1 to 7 in the sequence indicated by their order numbers (all goals were touched);

  • Stage 2: The volunteer performed the touch on the goals according to the following sequence: 1 to 7, 2 to 7 and 3 to 7. Goals 4, 5 and 6 were not touched because there was hidden in this stage;

  • Stage 3: The volunteer touched the goals in this sequence: 4 to 7, 5 to 7 and 6 to 7. In this stage, the goals 1, 2 and 3 are not touched because there were hidden at this stage.

Fig. 5
figure 5

Stages of the TUI interaction test (obtained from the interaction test of Sect. 4.3

Full size image

To obtain the data required for the calculation of the ACC, the following procedure was carried out. If the touch was made in accordance with the expected sequence, the numbers of positive attempts were increased (+1). Otherwise, the negative attempts were increased. It was not allowed to go to the next stage, if all the touches of the previous stage had not been completed. In each trial, the account of positive and negative attempts was reset. The totals attempts, positive and negative, were stored in a database file for further processing and analysis.

6.3 Testing GBUI

In this experiment, the software application implementing the GBUI was used. Unlike the previous case, the user did not touch the smartphone during the trials. The goals were gesture-goals, that is to say, the volunteer had to execute the indicated gesture in the sequence established in each stage. To identify the gestures, we used the armband MYO,Footnote 1 which is described in Fig. 6. MYO is a wearable device which measures the EMG signals and sends the raw data through Bluetooth to a fully computing device (computer or smartphone) for processing the data in order to recognize the gesture.

Fig. 6
figure 6

Armband MYO components

Full size image

MYO incorporates a total of eight electrodes of sEMG type. The sEMG electrodes have three contact zones with the skin, whose characteristics minimize the presence of induced noise, and their configuration on parallel bars, which surround the forearm, ensures consistency of the measurements [54]. No gel is required for its use. Further details can be found on the Web site of the manufacturer.Footnote 2 The IMU (accelerometer, gyroscope and magnetometer) was not used in the trials.

MYO was placed on the right or left forearm depending on whether the user was right or left handed. He/she was not allowed to change the armband from one arm to the other. The default calibration was maintained and the technique applied by [54] was used to avoid loss of calibration in each user.

The sequences were performed using the GCT (Fig. 7), which decomposes the sequences in seventh or fourth levels. The sequences performed by the volunteers were the following:

Fig. 7
figure 7

Stages and levels of the GBUI interaction test (obtained of interaction test of Sect. 4.3)

Full size image

  • Stage 1: The volunteer performs the start gesture (gesture-goal 1), then executes the option gestures from 2 to 6 and finishes with the action gesture (gesture-goal 7). All gesture-goals are executed;

  • Stage 2: The gesture-goals 1, 2, 3 and 7 are executed;

  • Stage 3: The gesture-goals 4, 5, 6  and 7 are executed.

The volunteer also receives notifications through audio messages for each gesture-goal. If the gesture was accepted, the “coin” sound was emitted. Otherwise, the “plop” sound was heard. In case of success (goal reached in the expected order), the smartphone emits an audio message in order to expect the next gesture-goal. This avoided the visual supervision of the mobile device, advancing in the sequence quickly and effectively. A maximum wait time of 10 s was set. If no gesture was detected in that period of time, and did not reach the last gesture-goal 7, the reading was canceled and a new sequence was started next. The calculation of ACC was similar to the TUI.

6.4 Results and discussion

Applying the experimental protocol described in the previous section, the relevant tests were carried out, in which participated each one of the volunteers. Subsequently, the comparative analysis of the two types of interface was performed using each anatomical positions provided in this investigation. The results are presented next.

6.4.1 TUI

Proceeding according to Eq. (1), we have been able to determine the ID for the TUI (see Table 3, where the values used for A and W are also included). As it is observed, the ID has a greater variation in stage 2 (Fig. 5), because the distance between each two goals demands more movement (e.g., the movement from goal 1 to goal 7 in stage 2 reached an \({{ID}}=3.94\)). The difficulty to reach a goal can increase when the smartphone has a larger screen size. Then, the larger separation of goals should be avoided in order to reduce the ID. On the other hand, we may have problems when the smartphone has a high resolution with a small screen size. In this case, the interaction points are too small to apply touch. Some solutions have been provided adopting implementations based on offscreen areas that can be visualized by the zoom utility [9, 63]. This also can indicate that triangular designs with vertexes at the top or bottom of the touch screen area should be avoided.

Table 3 ID reached by TUI
Full size table

In Fig. 8, we can see the average of attempts made by each volunteer: the positives or success (+) at the time of placing the goal and the negatives or errors (−) committed in each trial. The successes give a clear idea about of precision of the volunteers in normal (+) and special (+) position. For example, volunteer 5 had problems locating the goal in normal position. However, in special position achieved similar levels to other volunteers. In the case of the voluntary 15, it was observed that the opposite occured.

A slight decrease in the average number of attempts in the special (+) position relative to the normal (+) position may also be noted. This means that most users slow down the touch in special position. However, in the case of volunteers 3 and 6 quite the opposite occured, achieving a high degree of success in special position. Some volunteers reported experiencing fatigue in the hand that held the mobile device, and consequently discomfort to try of touch on the device in special position.

Fig. 8
figure 8

Positive versus negative attempts using TUI

Full size image

The average of positive attempts in the normal position reached 17.616 (\({\text {SD}}=2.131\)) in contrast to the average of the attempts in the special position that reached 17.48 (\({\text {SD}}=1.936\)). Thus, despite the fact that the volunteer in some cases was shown some instructions to improve the comfort of the interaction (taking the smartphone with one hand), the results did not improve. The average of negative attempts (−) reached 0.464 (\({\text {SD}}=0.377\)) in normal position and 0.632 (\({\text {SD}}=0.522\)) in special position. In this sense, some volunteers (2, 6 and 17) highlight in normal position and others (10, 11 and 13) in special position.

Regarding the ACC in the TUI in the studied positions, Fig. 9 shows a loss, although minimum, of ACC in the special position which reached 0.965 (\({\text {SD}}=0.029\)) compared to the normal position that reached 0.974 (\({\text {SD}}=0.021\)). Exceptionally, some volunteers reached an accuracy similarly high in the two positions. Some volunteers achieved absolute accuracy (i.e., volunteers 1, 4, 7 and 9), in both positions with this type of interface (see Figure 9).

Fig. 9
figure 9

ACC levels using TUI

Full size image

6.4.2 GBUI

Figure 10 shows the average number of attempts made by each voluntary when trying to reach the gesture-goal in the gestural interface, for each one of the positions studied. As in the previous case, the successes (+) at the time of placing the gesture-goal and the errors (−) committed in each test are shown. It can be seen that, in this type of interface, the volunteer achieved a greater number of successes at the special position, compared to the normal position.

Fig. 10
figure 10

Positive versus negative attempts using GBUI

Full size image

An average of 7.416 (\({\text {SD}}=1.696\)) was reached in the normal position, in contrast to an average of 7.8 (\({\text {SD}}=1.664\)) in the special position. In addition, we can argue that 60% of the volunteers improve the successes in special position, which is remarkable for the GBUI. As can be seen, the variation is minimum between one position and another. Perhaps the strongest advantage of this type of interface is that it does not depend on the distance or size of the importance zone (goal), and being independent of contact, we can assume that it adjusts their behavior to each user. This further indicates that the gesture fits much better to the needs of user interaction without causing excessive fatigue. The average of negative attempts (−) reached 0.84 (\({\text {SD}}=0.775\)) in normal position and 1.008 (\({\text {SD}}=1.038\)) in special position and remained low except for volunteers 7, 8, 14 and 16 in special position.

Fig. 11
figure 11

ACC levels using GBUI

Full size image

ACC levels of the GBUI are shown in Fig. 11. The ACC were very similar both in the normal position (\(ACC= 0.898\), \({\text {SD}}=0.095\)) and in special position (\(ACC= 0.885\), \({\text {SD}}=0.112\)). This indicates that GBUI is in some degree independent of the position (normal or special) taken by the volunteer to perform the gesture.

Fig. 12
figure 12

Comparison of ACC levels using TUI versus GBUI

Full size image

For several volunteers (i.e., 4, 5 and 12), the improvement in accuracy, in special position, was very evident. In other cases, the accuracy remained very similar, perhaps independent of the anatomical position of the volunteer. The majority of volunteers had no major problems with this type of interface and they adapted straightforward to the use of gestures. Only volunteer 9 was able to reach absolute levels of accuracy in both positions, while in contrast, only volunteer 14 presented a significant loss of accuracy in special position.

6.4.3 Comparing TUI versus GBUI

We try to give a general picture of the behavior of the analyzed interfaces. Figure 12 allows to observe the accuracy behavior in both the TUI and the GBUI. Each interface is compared according to the anatomical positions analyzed in this paper, that is to say in normal position as in special position.

Figure 12a shows that the accuracy achieved by volunteers using the GBUI at the normal position is very similar in many cases and, in some cases (although a few), exceeds the accuracy achieved with the TUI, during the trials performed. A notable variation of accuracy (below 0.9) could also be observed in volunteers 4, 5, 6, 7, 8, 11, 12, 14 and 16 when using the GBUI, because for them the TUI was more precise. The other volunteers seem to be appropriately adapted to the use of the GBUI in the normal position while always maintaining a relative similarity. The TUI reached an average accuracy of 0.974 (\({\text {SD}}=0.021\)), while the GBUI reached an average accuracy of 0.898 (\({\text {SD}}=0.095\)) in this position.

Figure 12b shows how the GBUI outperforms or matches the TUI in special position (volunteers 9, 10, 13, 17, 18, 19, 22 and 24). There are exceptions (below 0.9) such as the volunteers 3, 5, 6, 7, 8, 11, 14, 15 and 16, in which the GBUI shows a decrease of accuracy. However, the other volunteers show a better coupling with the gestural interface and even improve their accuracy in the special position (volunteers 10, 13, 17, 18, 19, 22 and 24). The TUI reached an average accuracy of 0.964 (\({\text {SD}}=0.029\)) while the GBUI reached an average accuracy of 0.885 (\({\text {SD}}=0.112\)) in this position.

7 SICLLE: emergency call control interactive system

Fig. 13
figure 13

System architecture of SICLLE

Full size image

The analysis carried out has led to the conclusion that the GBUI is similar to the TUI for the management of phone calls. Thus, an application with a GBUI may be very suitable. With this aim, an application for mobile devices named SICLLE (emergency call control interactive system) has been implemented using the GBUI described in Sect. 5. This application can be particularly useful for people who require gestural interaction for convenience or for any special needs (e.g., illness, physical limitation, visual problems).

7.1 GBUI architecture

Figure 13 shows the architecture used to implement SICLLE. This architecture can be applied to develop any mobile application for phone calls with GBUI and gesture recognition through EMG signals. Its design is based on layers and is described below:

  • Data layer This layer is responsible for capturing the user gestures. In SICLLE, this layer is instantiated using an armband MYO. This gestural capture device captures the gesture through sEMG electrodes. This layer sends data to the upper layer using the Bluetooth protocol with low power (BLE) and faster information exchange;

  • Application layer This layer is in charge of the interpretation of gestures and processing of commands. This layer comprises the smartphone or mobile device (MD) which receives the data sent by the under layer. In SICLLE, a midrange smartphone with Qualcomm quad-core processor MSM8926 of 1.2 MHz, with 1 Gb RAM, 8Gb flash memory, compatible with Bluetooth 4.0, OS Android 5.1 was chosen. This device conforms to the requirements of size, weight and resolution set in Sect. 4. The MD receives data sent by data layer. Then, the data are processed through several stages. After applying a gesture recognition process, the MD identifies a valid gesture comparing it with all the gesture patterns stored in a Gestural Dictionary (GD). Once the gesture is identified, a command will be associated from the Command Dictionary (CD) in order to be executed in the MD during the call. In addition, a voice message library (VL) has been implemented. Its main objective is to store the audio messages to notify the users of the selected action (option). Several voice messages inform the user of the chosen action as well as any other in progress.

  • SP layer It consists of the service provider (SP) of mobile telephone. The SP is the one responsible for establishing the call using the phone network, either fixed or mobile, and takes into account additional variables, such as balance, cost and protocol,, to provide the required service.

7.2 Commands and gesture library

The commands implemented in SICLLE through gestures are described in Table 4 (similar gestures have been used in [47, 54, 56, 65, 68]). Consequently, every gesture is a command which is associated with a specific action. These actions allow the management of phone calls in a smartphone.

Table 4 Gestural commands used in SICLLE
Full size table

SICLLE, once executed, acts as a conventional terminate-and-stay-resident program (TSR) in device memory waiting for the opening gesture. As soon as the interaction is initiated the software is awaiting for a gestural command for call management or phone book (SICLLE has its proper phone book).

7.3 SICLLE design

The design of an application based on GBUI requires to resolve two issues: (a) to set the gesture control in order to specify how the recognized gestures are managed and (b) to design the views using traditional TUI that are executed in smartphone in order to configure the application.

The user interaction with gestures must be performed according to the gestural control tree (GCT) shown in Fig. 14. GCT indicates the valid sequences of the gestures that are allowed by the GBUI of SICLLE.

Fig. 14
figure 14

The gesture control tree (GCT) defined for the SICLLE system

Full size image

SICLLE includes also some views using traditional TUI to facilitate its configuration. Figure 15 shows some screens used to modify the configuration of the system, as described in the following.

Fig. 15
figure 15

SICLLE screenshots

Full size image

Find device This option allows the user to link the GCD device with the MD. If there are several gesture capture devices, only one will be associated at a time. The application enables Bluetooth communication with the mobile device if required.

Setup It allows the assignment of a gesture to an action or command, taking into account those gestures implemented and included in the gestural dictionary. The user can personalize his/her gestural interaction.

Management This option allows access to the phone book of the smartphone, which is shared with the phone book of SICLLE. The user can add, modify and delete contacts in both phone books. The appropriate configuration allows the assignment of a contact number as preset for a call. The photo of the contact is displayed, if available.

Help It shows the set of gestural commands (gestures) available in the application for the users into the screen of the smartphone. A brief description of the assigned action to each command is also given. Voice messages are related to each action.

7.4 Validating SICLLE

With the aim of validating SICLLE, a new experiment was conducted. Thus, the user experience of the smartphone with the described GBUI was analyzed while the users utilized SICLLE to make phone calls. All the volunteers who participated in the previous study participated in the validation as well (25 in total). With the aim to facilitate user learning, the gestures used were the same as those utilized in the interaction test for GBUI (Sect. 4). The GCT implemented in SICLLE (Fig. 14) also shows the command associated with each gesture.

7.4.1 Experimental protocol

During testing, SICLLE worked adequately, allowing users to manage phone calls through a gesture-based interface without any physical contact. Two types of phone calls (the two cases described in Fig. 14) were considered: (a) calls to a preset contact and (b) calls using the phone book. In the case of (b), four contacts were added to the phone book of each participant and the participant was unable to see them until the search was required. Moreover, users were free to place and move as they wished. Ten trials were performed by each user, five of type (a) and five of type (b). For both tests, the GCT shown in Fig. 14 was used.

To call a preset contact (a), the time limit was 20 s. No error was made in this type of calls. The average time to complete this task was 10.9 s. (\({\text {SD}}=1.05\)). In certain cases, the telephone service provider was responsible of time delays. The time was measured until the phone tone sounded, because in this moment, the user finished the call with the corresponding gesture (goal 3).

To call a contact in the phone book (b), the name of the contact was given. Then, the user had to find the contact in the phone book and call him/her. The navigation in the phone book was performed using the GCT shown in Fig. 14. The average time to complete this task was 14.46 s (\({\text {SD}}=0.66\)).

It should be noted that some volunteers experienced problems of gesture recognition at the beginning of the interaction. This was due to the physical and anatomical conditions of their body, especially of the muscular development of the forearm of each person. This caused loss of calibration of the GCD device. To solve the problem, the recommendation given by [54] was used. All trials started when the correct response of the GCD device was confirmed.

7.4.2 Results and discussion

After using the software, the users were interviewed about their experience with the use of SICLLE. All of them (100%) expressed greater level of comfort in not having to hold the mobile device, particulary in special positions (for instance, in a lying position). In addition, 19 participants (76%) emphasized the difficulty of using the smartphone in a lying position. All users (100%) agreed that SICLLE can be very useful in case of problems of mobility or vision.

Additionally, several questions on the facility of learning (questions 1–3) and subjective satisfaction (questions 4–6) on SICLLE were made. These questions are shown in Table 5, and the users responses are summarized in Fig. 16.

Table 5 Questionnaire use of SICLLE
Full size table

According to the answers of the users, it is possible to argue that: (Q1) 56% of users consider that SICLLE is easy to learn and use to manage telephone calls; (Q2) 80% found that it was easy to interact using levels in SICLLE; (Q3) 76% manifest that it was easy to use the same gesture in different SICLLE options; (Q4) 64% consider that the use of SICLLE has been more agreeable in contrast to the traditionally app; (Q5) 92% consider the aid by voice appropriate in SICLLE; (Q6) 100% state that they would agree to install SICLLE on their smartphone (for occasional and emergency use, especially).

Fig. 16
figure 16

User responses to the questionnaire

Full size image

These results seem to indicate the viability of SICLLE and hence also support the gestural interaction characteristics implemented in the proposed GBUI. This form of natural interaction could be incorporated as a multimodal alternative to make phone calls in smartphones, which will benefit users with some restrictions or disabilities (Parkinson’s disease, paralysis, problems in motor skills, visual difficulties, convalescence, etc.) but also to any user in certain situations (emergency, screen break, a special anatomical position, etc.).

8 Conclusions and future works

Human–computer interaction is transformed as well as the technology used to achieve it. Thus, the technological advances in natural interaction are enriching the user experience in very diverse applications. The gesture as a natural user interface has proved its competence in different contexts to control and manage devices. The present paper has shown the ease of use of a classic TUI in contrast to a new proposal of GBUI in smartphones. With this aim, two main contributions are made: the comparative analysis of both interfaces and an application for management of phone calls in smartphones using GBUI.

The analysis of the interfaces has been performed in two basic positions, very common among users of smartphones, which are “normal” when the individual is standing and “special” when the individual is lying down. The first position represents a common position for healthy individuals, while the second is used to represent sick, disabled or comfortable individuals. This analysis evidences the high accuracy which the natural user interfaces can have based on gestures compared to touch screen user interfaces. To achieve a valid estimation, a test was performed with 25 participants who performed an interaction test structured in three stages using two software applications: the first, based on TUI interaction, and the second substantiated on GBUI interaction.

The results obtained indicate that the GBUI achieves an average ACC of 89.8%, in contrast to the higher average accuracy achieved by the TUI, which reached 97.4% in the standing position. Conversely, in the lying down position, GBUI achieved an average ACC of 88.5%, while TUI reached an average ACC of 96.5%. The GBUI average ACC percentages, although not comparable with the high accuracy rate achieved by the TUI, are very reasonable giving a good response, independently of the anatomical position of the user (only 1.3% difference between the two positions). On the other hand, GBUI has additional properties to TUI, which can be more convenient in specific contexts. GBUI, by definition, is independent of the touch (goal) area and does not require any contact during the interaction, which suggests that users with visual impairments could gain many benefits by using it in a mobile application. In addition, the GBUI based on the proposed architecture has unique advantages that only contactless interaction (using sEMG and Bluetooth communication) and free hand gestures can offer. Accordingly, in GBUI there is no occlusion, it is independent of the luminosity, it releases the user from visual tracking and offers great possibilities of accessibility.

During the study, it was also determined that the ID of a mobile device, keeping constant the size of the touch area (goal) varies in direct relation with the required touch distance. The ID found with \(A=20\) mm and \(W=11.5\) mm, reached 1.80. However, at longer distances (between 50 and 55 mm) the ID increases to 3.12, and when the distance is increased to 88 mm, the ID reaches a value of 3.94. This also suggests that, at a greater distance, the ID increases, even though the size W of the goal is kept constant. This shows that the index of difficulty to reach a goal can increase according to a larger screen size, as, for example, may occur in tablets. In our specific case, using a smartphone of medium range (\({\text {height}}=139.9~{\text {mm}} \times {\text {width}}=71.1~{\text {mm}}\)), the ID is in the order of magnitude of between 1.80 and 3.94. In this sense, a significant advantage of the GBUI is its independence of the distance.

SICLLE, an emergency call control interactive system, has been developed using a layered architecture specific for GBUI in mobile devices proposed in the present paper. SICLLE is a simple application to manage calls and phone book in compatible Android smartphones. SICLLE implements the concept of gesture control tree to achieve an effective gesture-based interaction through gesture-goal (gesture associated with interaction point). The GCT implemented in SICLLE allows to reuse the same gesture at different levels of the GCT and facilitates the tracking of the sequence of gesture-goals in GBUI. In addition, it enriches the alternatives of use of the gestural library implemented in the mobile device. To measure their degree of satisfaction and use, a validation test was performed. Twenty-five users participated in the trials using SICLLE to make direct calls (preset calls) and calls using the phone book. The obtained results show that the users expressed greater level of comfort (64%) in not having to hold the mobile device with one hand to interact with it, particularly in special positions (for instance, in a lying position). In fact, 19 participants (76%) emphasized the difficulty involved in using the smartphone with traditional TUI in lying position. According to the answers about SICLLE, 56% of users consider it was easy to learn and use to manage telephone calls, while 80% found that the interaction by levels was easy to learn; for 76% it was easy to use the same gesture in different options; 64% argued that the use of SICLLE was more agreeable in contrast to the TUI interaction; 92% consider the aid by voice appropriate and timely; finally, 100% would consent to installing SICLLE on their smartphone. This suggests that SICLLE can provide to users an alternative for managing their phone calls. In that sense, all users (100%) agreed that SICLLE can be very useful in case of problems of mobility or vision.

Note that the GBUI architecture presented in this paper can be extensible to other kinds of mobile applications, as well as the results of the comparison between GBUI and TUI can be used as a reference in other contexts, even for people with special needs. Thus, all these facts seem to indicate that technology advances in gestural interaction at a steady pace and multimodal interaction opens new expectations. This allows us to imagine a great variety of interaction possibilities by means of gesture, using wearable devices that incorporate computing capacity, mobility and multiple communication protocols (among others).

As future work, we intend to expand the gestural library, using techniques of machine learning, which could be adopted as a standard control gestural language. An additional objective is to apply innovative concepts such as the CGT to other types of gesture (body, facial, etc.) to explore the full potential of the GBUI interaction. In addition, new sensors could be incorporated in combination with sEMG (e.g., iR) to enhance the user interaction. In parallel, we intent to propose other measures (complementary to ID and accuracy) to evaluate the effectiveness and precision of the gestural recognition in GBUI. Finally, it is also a challenge to develop other architectures that allow multi-protocol communication, as well as, to implement real-time control gestural interfaces for home (e.g., multimodal games) and for industry (e.g., robotic control).