1 Introduction

Augmented reality (AR) allows users to see virtual content with a live real-world view and gives users a chance to interact with that virtual content in real time. Recent advancements in handheld devices have made handheld AR very popular. Handheld devices have been identified as the most promising devices for AR [3, 9]. Current tablets and smartphones are lightweight and truly mobile. All the necessary hardware for AR is packaged into one device, such as high resolution display, camera, graphics, GPS compass and accelerometer [21]. Handheld devices give video see-through AR experiences via powerful processors and various built-in sensors [7]. By using handheld devices, users can see virtual objects on top of the real-world view on the screen.

The handheld AR has important applications in fields such as gaming [11], interactive marketing and advertising [7], education [16], and navigation [20]. However, some major technical obstacles must still be overcome before AR can realize its true potential. One of the major problems is allowing users to interact with a virtual object in an augmented scene with a handheld device [7].

In general, the interaction through handheld devices is known as magic lens because users see the real world with virtual content through a display on the device by using a camera on the opposite side of the device [7, 12]. A variety of interaction techniques are available for interaction (manipulate, translate, rotate) with these virtual contents. A handheld AR interaction technique can be divided into two categories. Different researchers have classified it in different ways. According to Bai et al. [5], a handheld AR interaction uses tangible and intangible interaction methods. Bai et al. [5] defined a tangible interaction as a physical interaction with a device, and an intangible interaction as an interaction by means of a gesture, gaze or speech. Gervautz and Schmalstieg [7] classified a handheld AR interaction technique as an embodied and tangible interaction method. An embodied interaction is based on the interaction between the movement of a device and the touch screen in the AR scene, while a tangible interaction is the direct manipulation of a known physical object, such as the movement of a marker or hand gesture recognition. However, such different classifications and definitions may confuse readers. To make it clearer, handheld AR interaction techniques have been classified into two types. These are device-centric interactions and natural gesture interactions.

Device-centric interactions are those that use the device itself to interact with virtual contents in an AR scene such as motion sensor, device movement, touch screen gestures and keypad input. By using a tangible handheld device, users are able to manipulate virtual content. For example, a touch screen gesture can be used for scaling the virtual object on a handheld touch display. On the other hand, a natural gesture interaction allows users to use their finger or hand gestures to interact with a virtual object in an AR scene. The user finds that gestural interaction is more fun and engaging, but research has shown that gestural interaction is less suitable for handheld devices [10]. Gestural interfaces on handheld devices also suffer from the more limited distance between the camera and the interacting appendage, which is not an ideal combination [23]. The device-centric interaction for handheld AR is explored in this work.

The touch screen input is one of the common device-centric interaction methods in handheld augmented reality. Most of the handheld devices in the market come with a touch screen interface in place of traditional inputs such as physical buttons. However, precise interaction with a touch screen is not easy due to the shaky view. To overcome this problem, researchers have proposed a few techniques. One novel technique is the freeze interaction technique, which allows users to freeze their AR view and interact with the virtual content. The freeze interaction technique shows its strength in a shaky view, but the problem is when users freeze their view, it freezes the whole scene and the real-world view from the camera is not being updated. When the virtual object is updating continuously, the real-world view remains as a still picture until the user unfreezes the scene. This may be a problem when users have to deal with the current up-to-date real-world scene. While working in a frozen scene, the physical world is not updated in the AR scene. Therefore, when users unfreeze the scene, they can lose the view that they last had on the live AR scene.

In this work, the freeze interaction technique was explored and the ‘Freeze-Object’ interaction method was implemented for the handheld AR environment to overcome the existing problem. The Freeze-Object interaction technique will allow users to interact with a virtual object with a live updated real-world view. The rest of the paper describes the related work, interaction concept, implementation details and two user evaluations of the Freeze-Object interaction technique with the existing freeze technique.

2 Related work

A handheld AR interaction metaphor is different from a traditional mobile AR interaction [9]. Handheld devices do not use HMD. A touch screen interface is common in Smartphones and is also widely used in current handheld AR environments. However, touch interaction is not easy in handheld AR. It causes a shaky view because users have to hold the device with one hand and touch the interface with the other hand [4, 15]. To observe a large environment, users have to move the device frequently, which causes marker tracking failure [12, 14]. This problem can be solved by the “Freezing” method.

Researchers have explored the freezing method in several applications. Abawi et al. [1] implemented a freeze interaction method for applications in oil refineries to overcome jitter problems with the ARToolKit and bed light conditions in refineries. In their application, the freeze method was very useful for the authoring process. Güven et al. [8] proposed a set of freeze interaction techniques for authoring situated media. These techniques are the Freeze-Frame, Freeze-n-Link, Freeze-n-Move and Freeze-n-Overlay. The ‘Freeze-Frame’ allows the user to capture a snapshot (frame) of the environment to work on and later map the results back to the physical world. The ‘Freeze-n-Link’ allows users to take a snapshot of several frames, and is used to create links between a location and an object in different frames. The ‘Freeze-n-Move’ allows users to move a group of objects to a new location while maintaining their organizational integrity. The Freeze-n-Move technique was inspired by Robertson’s Fix and Float technique [19], whereby a 3D object becomes fixed for the user’s view and is attached to the screen. The user can then move the object using egocentric navigation to the target location before releasing it. The Fix and Float technique was built for a virtual environment and it cannot be used in AR. Guven et al. [8] extended the Fix and Float technique to include a freeze approach in an AR environment. Another freeze technique by Guven et al. [8] is the ‘Freeze-n-Overlay’ technique, which allows users to take a picture of the original scene with the virtual object of interest and overlay a semi-transparent layer of it on top of the target scene. This technique makes it possible to position the virtual object onto a desired destination while maintaining their original context. Lee et al. [15] further investigated the usability of the freeze interaction technique through a formal user experiment and implemented the ‘Freeze-Set-Go’ or FSG interaction technique in handheld AR. They explored marking and adding handwritten annotation in the AR scene. In their case, the interaction in a freeze mode is much more comfortable for the annotation task because the user can make annotations in a comfortable posture rather than having to keep holding the device until the task is completed. Their user experiment showed that by using the FSG, the user can interact with the AR scene more accurately, and it works more effectively in cases where the user has to work in difficult conditions. Boring et al. [6] presented the ‘Touch Projector’ system to interact with a remote screen using live video images on their mobile device. However, handheld devices do not provide adequate stability and control for precise manipulation. To overcome this problem, Boring et al. [6] used the freeze approach to improve their system. Their user study showed the highest performance was achieved by participants using the freeze interaction method. Langlotz et al. [13] used the freeze method in their in-situ content creation application for an accurate touch screen interaction. In their application, once the user is in position, the view can be frozen by touching the button on the touch screen, and the task of authoring can begin. After the task has been completed, the user can unfreeze the view. Bai et al. [5] utilized the freeze interaction technique to overcome the screen shaking problem while interacting with the touch screen of the smartphone, and implemented the ‘freeze-view-touch’ interaction for manipulating virtual 3D objects in an AR scene. User experiments have shown that users perform the manipulation task faster and more accurately by using the freeze-view-touch technique.

However, current freeze interaction approaches have drawbacks [14, 15]. When a user freezes the AR scene, the real-world view is also frozen. When virtual objects are updating continuously, the real-world view remains as a still picture until the user unfreezes the scene, thus reducing the real-time AR experience, making it not sufficiently attractive for users [5]. This will be the problem when users have to deal with current up-to-date real-world scenes, such as when users are walking while using the AR application or need to deal with different real-world backgrounds. While working on a frozen scene, the physical world is not updating in the AR scene. Therefore, when users unfreeze the scene, they can lose the view that they last had of the live AR scene. Also, more time is required if the user wants to deal with different real-world backgrounds because they have to unfreeze the scene each time and change the AR tracking target position before tracking once again. To overcome the current problems in the freeze technique, this work proposed an improved freeze interaction technique known as the ‘Freeze-Object’ for handheld AR systems. The Freeze-Object allows users to freeze the virtual object while the real-world view remains updated.

3 Freeze-object interaction technique

The main purpose of this work is to develop an improved freeze interaction technique known as the ‘Freeze-Object’ for handheld AR environments to avoid frozen real-world backgrounds and to offer more precise interaction to users. The Freeze-Object supports three touch gesture manipulation methods, namely translation, rotation and scaling. The Freeze-Object interaction technique stops the tracking thread and renders the virtual object on the screen in a live video background, where users are no longer required to do the visual tracking. After activating the freeze mode, users are able to select the manipulation mode. The detailed process within the framework is illustrated in Fig. 1.

Fig. 1
figure 1

Freeze-object interaction technique framework

Full size image

The Freeze-Object method works as follows: The user holds the handheld device in an appropriate position with a clear view of the virtual object. When the camera on the device identifies and detects the trackable target correctly, the user clicks on the “Freeze” button to freeze the virtual object. A virtual object will only freeze upon successful visual tracking. If the visual tracking fails, the freeze button will not be activated. Once the freeze button is pressed, the system will freeze a virtual object on the display screen. After freezing the object, users do not have to point the handheld device at the target for tracking, and they can move the device in any direction to see the virtual object in different real-world backgrounds.

After freezing the virtual object, users can perform touch manipulation by selecting the manipulation mode from the screen menu. Figure 2 shows the AR application interface, which consists of six buttons: Freeze, Move (translation), Rotation, Scaling, Reset and Quit. The screen menu is chosen to reduce the user’s finger movement and task time. Users are able to select one manipulation mode at a time. Once users have completed all the manipulations, they can click the freeze button again to unfreeze the virtual object on the tracking target.

Fig. 2
figure 2

Application interface

Full size image

3.1 Implementation details

The AR system was built on the HTC One X Plus smartphone. It has a 1.7 GHz quad-core processor, 1GB RAM and ULP GeForce 2 GPU. The device is running on Android OS 4.2.2. It has a 4.7-in. Super LCD2 capacitive touchscreen, which supports multi-touch gestures, and an 8-megapixel camera capable of recording 1080 p hd videos.

To implement the Freeze-Object interaction technique in handheld AR, various software development tools were used. The Unity 3D [22] with Android SDK [2] was used to develop a handheld AR application. The Unity 3D is a cross-platform game engine that supports intuitive tools for interactive 3D content. The Android SDK provides the API libraries and tools for developing Android applications. The AR tracking was implemented using QUALCOMM Vuforia vision-based SDK [18]. The Vuforia AR tracking provides robust vision-based tracking and fast computation time. The 3D object rendering and graphics were based on Unity 3D, which has a powerful rendering engine fully integrated with a complete set of intuitive tools [17]. It uses the OpenGL ES library for rendering objects in a mobile platform. Figure 3 illustrates the software components of the AR system.

Fig. 3
figure 3

Software components of AR system. The main components are the Vuforia tracking library, Unity 3D game engine for rendering and Android SDK for using android OS running touch screen handheld device

Full size image

4 First user study

A user experiment was conducted to evaluate the improved freeze interaction technique known as ‘Freeze-Object’. In the user experiment, the Freeze-Object interaction technique was compared with an existing freeze technique similar to the FSG interaction technique [15] in a handheld AR environment. The user study was focused on investigating the user’s task performance and the user’s preference in terms of usability of the interaction technique.

4.1 Experimental setup

The user study was set up using a within-group factorial design. The independent variable was the interaction technique and the dependent variable was the task completion time. There were two different touch screen AR interaction techniques: one was the ‘Freeze-Object’, the improved freeze interaction technique, and the other was the existing freeze interaction technique, with three basic manipulation tasks with subtasks. Each participant had to perform several manipulation tasks using both interaction techniques. Furthermore, the users’ preferences for the interaction techniques were measured in terms of usability by means of usability questions.

The study began with the distribution of the consent form regarding the experiments, and the users were asked to complete the questionnaire regarding their background, such as their age, gender, educational level and experience with using a handheld touch screen device and with the AR interfaces on the handheld device. A brief introduction and hands-on training were provided about the AR interaction techniques and detailed instructions were given concerning the testing environment. Next, the users were given time to practise with both AR interaction techniques. The subjects were encouraged to take all the time they needed to practise with the AR interface. When they felt comfortable about controlling the interface, the test session began. Once they started the test, they were not allowed to ask for any help. Upon completion of the task using each interaction technique, the participants were asked to fill out a pre-experiment questionnaire (Table 1) using a five-point Likert scale for each manipulation (translation, scaling and rotation) for both techniques.

Table 1 Pre-experiment questionnaire. The given interaction technique was: (for translation, rotation and scaling)
Full size table

At the end of the evaluation, they were given a post-experiment questionnaire in which they were asked to rank the interaction techniques and why they chose that particular technique. Each participant took approximately 45 min to complete the whole experiment.

4.1.1 Participants

A total of 14 participants (11 males and 3 females) were recruited for the experimental user study from within the university. All of them were postgraduate students with Information Technology backgrounds. Their ages ranged from 22 to 33 years. Twelve participants were right-handed and 2 were left-handed. During the experiment, 12 users held the handheld device with their left hand, while the other 2 held the device with their right hand. No significant difference was observed regarding handedness. All 14 participants had previous experience with using the handheld touch screen device and 12 of them used it every day. Among the 14 participants, only 6 of them had little experience with using an AR interface in the handheld touch screen device, while 8 of them had no previous experience with using a handheld AR interface.

4.2 Experimental task

The experimental task involved three basic gesture manipulations (translation, rotation and scaling) of a virtual object (a 3D cube) using the Freeze-Object and the existing freeze interaction method in a handheld AR environment. Each participant was asked to perform all three manipulations using the two interaction techniques. The experimental task was to manipulate a virtual object and match it to the indicated position or size. For all the manipulation tasks, the indicated target and virtual object were clearly displayed on the screen of the device. The task was timed after the selection of the mode of manipulation. The participants were asked to perform the task as fast and as accurately as they could.

In the translation task, the participants had to move the virtual 3D cube in three different directions (left, right and up). The translation target was set up in the physical world. The participants had to view the target through the camera of the handheld device. In both interaction techniques (the Freeze-Object and the existing freeze method), the task was the same. However, the performance of the task was different. In the Freeze-Object interaction technique, the participants were asked to move the device for the translation task, while in the existing freeze interaction technique they were asked to do so with touch gestures. The Freeze-Object interaction technique allows users to freeze the virtual object, which stays in position while the background is live, and this helps users to see the virtual object in a different physical background without visual tracking. In the existing freeze method, the participants had to freeze the AR scene and keep moving the virtual object by dragging it on the touch screen. To see the target on the screen, the participants had to keep all three targets within view of the camera and freeze the AR scene (Fig. 4).

Fig. 4
figure 4

a participant performing translation task using Freeze-Object interaction method, b participant is using existing freeze method for translation task

Full size image

In the rotation task, the participants had to rotate a virtual 3D cube clockwise around the Z-axis at angles of 45, 180 and 330° from the initial position using one finger. The rotation was set up in one axis to avoid interaction complexities because when users perform the rotation using the touch screen interface, it will lead to deformable transformation in several axes. Figure 5 shows a participant performing the rotation task in the Freeze-Object interaction technique.

Fig. 5
figure 5

Participant performing rotation task using one finger around Z-axis

Full size image

In the scaling task, the participants had to change the size of the virtual 3D cube along all three axes together to 0.5 times smaller, and 1.5 and 2 times larger from the initial position. The pinch-to-zoom gesture (using two fingers to zoom in and zoom out) (Fig. 6) was used for scaling the virtual object. The system had the same working principle as rotation. The virtual object was reset to its initial position after having completed each subtask.

Fig. 6
figure 6

Participant performing scaling task using pinch-to-zoom gesture

Full size image

4.3 Experimental results

This section presents the results of the user study. As mentioned earlier, the main aim of the user study was to measure the user’s task performance and preference of the interaction technique. The task performance was analysed based on the time taken by the user to complete the task during the tests. The user’s preference of the interaction technique was analysed based on the usability questionnaire feedback.

The following section describes the task performance results of the two interaction techniques, while the next section describes the subjective feedback from the participants.

4.3.1 Performance analysis

The participant’s performance was analysed based on the time taken to complete the task using both interaction techniques. In both interaction techniques, the participants performed three tasks in a handheld AR environment. These tasks were translation, rotation and scaling. The system automatically recorded the time taken to complete the rotation and scaling tasks, while the translation time was recorded manually because the translation task was different in both interfaces. In the Freeze-Object, participants had to move the device to perform the translation, while in the existing freeze technique they had to do so using touch gestures. While moving the device, it is not appropriate to measure the time automatically. The objective performance was measured according to the time taken to complete each task.

A paired sample t test with alpha p = 0.05 was performed in order to analyse the performance of the participants and to determine the significance between two conditions. The result of the paired sample t test (Table 2) showed that there was a significant difference between the translation tasks in both conditions. In the Freeze-Object method (Translation1, M = 6.9721, SD = 2.79108) and in the existing freeze method (Translation2, M = 17.6386, SD = 3.74636); t (13) = −12.157, and p = 0.00. The results showed that the performance of the translation task using the Freeze-Object was significantly faster than the existing freeze method. No significant differences were found in the performance of the other two tasks (rotation and scaling). However, for the overall task completion time, the Freeze-Object interaction performed faster than the existing freeze technique. Figure 7 shows the average task completion time for each of the three types of tasks for both conditions, as well as the overall average time for all the tasks.

Table 2 Paired sample t test
Full size table
Fig. 7
figure 7

Mean completion time for overall and each task

Full size image

4.3.2 Preference analysis

In addition to the participant’s task performance (task completion time), the participant’s subjective feedback was collected using usability questionnaires. Each participant was asked to answer the six questions listed (Table 3) after performing each basic manipulation task (translation, rotation and scaling) in both conditions. After the whole experiment was finished, the participants were asked to give further comments on the interaction techniques, and rank which interaction technique they would prefer if they were required to perform similar tasks in the future. All these questions were answered on a five-point Likert-Scale (with 1 indicating strongly disagree, and 5 indicating strongly agree). A Wilcoxon Signed-Rank Test was performed for the significant differences between the two conditions, using Z statistics and p < 0.05. This test was chosen because data from the questionnaires were not normally distributed. The subjective result (Table 2) showed that there was a significant difference in the translation manipulation task in terms of (Q1) easy to learn (Z = −2.251, p = 0.024 [p < 0.05]), (Q2) easy to use (Z = −2.489, p = 0.013 [p < 0.05]), (Q3) useful to complete the task (Z = −3.213, p = 0.001 [p < 0.05]), (Q4) not mentally stressful (Z = −2.714, p = 0.007 [p < 0.05]), and (Q6) engaging (Z = −2.739, p = 0.006 [p < 0.05]). Users felt that for the translation task, the Freeze-Object interaction technique was easier to learn, easy to use, useful to complete the task, not mentally stressful and more engaging compared to the existing freeze method. There was no significant difference for (Q5), not physically stressful (Z = −1.098, p = 0.272 [p > 0.05]). Users felt the existing freeze technique was less physically stressful than the Freeze-Object interaction technique.

Table 3 Wilcoxon signed-rank test
Full size table

No significant difference was found in the rotation manipulation task for all six questions (p > 0.05 for all responses). The users felt the rotation task was easier to learn, easy to use, useful to complete the task, less mentally stressful and less physically stressful in the existing freeze technique compared to the Freeze-Object interaction technique. It was found that there was a significant difference in the scaling manipulation task in terms of (Q1) easy to learn (Z = −2.236, p = 0.025 [p < 0.05]) and (Q6) engaging (Z = −2.121, p = 0.034 [p < 0.05]). The participants felt that the Freeze-Object interaction technique was easier to learn and was engaging compared to the existing freeze technique. However, no significant differences were found in the other four questions for the scaling task (p > 0.05 for all responses). The users felt the existing freeze technique was easier to use, useful to complete the task, and less mentally and physically stressful compared to the Freeze-Object technique.

At the end of the questionnaire, the participants were asked to choose one interaction technique for each manipulation condition and to state their reason for choosing the technique should they be required to perform similar tasks in the future. For the translation task, all the 14 participants chose the Freeze-Object interaction technique because they felt that it was easy to move the device rather than to use touch gestures, and at the same time it gave them a real-time AR experience. A real-time AR experience allows users to see virtual objects with a live real-world view. For the rotation task 8 participants chose the existing freeze method, as they found it hard to rotate the virtual object while holding the device, and 6 chose the Freeze-Object technique. For the scaling task, 8 participants chose the existing freeze method and 6 participants chose the Freeze-Object interaction technique. However, for the overall task, 13 participants selected the Freeze-Object interaction technique because while manipulating the virtual object they could see the real-time updated background. They mentioned that it was easier to see the virtual object in any background and it was easy to use. One participant chose the existing freeze method for the overall task. The users’ preference for the interaction technique can be seen in Fig. 8.

Fig. 8
figure 8

Users’ preference for similar task

Full size image

5 Second user study

A second user study was conducted to evaluate the “Freeze-Object” interaction technique in a real application scenario. In this experiment, the “Freeze-Object” technique and the existing freeze technique were applied to an interior design application, where users had to manipulate a virtual table lamp. In this experiment, both interaction techniques were compared in terms of user performance and user preference.

5.1 Experimental setup

The second user study was set up with a group factorial design. The independent variable was the interaction technique and the dependent variable was the task completion time. Users performed three manipulation tasks (translation, rotation and scaling) using the “Freeze-Object” and existing freeze interaction technique in an interior design application. Furthermore, the user’s preference for the interaction technique was measured using a usability questionnaire.

At the start of the study, the background information of the users was obtained, such as their age, gender, education, experience with using a handheld touch screen device and an AR interface in the handheld device. A brief introduction about AR and hands-on training was provided for practical tasks. Next, the users were given time to practice both AR interaction techniques. When they felt they were ready, the actual testing began. Upon completion of the task using each interaction technique, the users were asked to answer a pre-experiment questionnaire (Table 1) using a five-point Likert scale. At the end of the study, the users were asked to rank the interaction technique in a post-questionnaire for each manipulation task and to state their reason for choosing a particular technique. Each participant took approximately 30 min to complete the whole experiment.

5.1.1 Participants

A total of 8 participants (6 males and 2 females) took part in the second experiment. All of them were postgraduate students recruited from within the university. Their ages ranged from 24 to 30 years. Six participants were right-handed and 2 were left-handed. No significant difference was observed with regard to handiness. All the participants had previous experience with using a handheld touch screen device every day. Among the 14 participants, only 6 of them had some experience of using an AR interface in the touch screen device, while 2 participants had no previous experience.

5.2 Experimental task

In the second user study, the “Freeze-Object” interaction technique for interior design applications was evaluated by comparing it with the existing freeze technique. In the experiment, the users had to detect the target image, and manipulate a virtual table lamp and match it to the indicated size or position. The manipulation tasks included translation, rotation and scaling. For all the manipulation tasks, the indicated target and virtual object were clearly displayed on the screen of the device. The tasks were timed after the selection of the manipulation mode. The participants were asked to perform the tasks as fast as they could.

In the translation task, the participants had to move a virtual 3D table lamp in two different directions. The translation target was set up on the physical world. The participants had to view the target through the display on the device. The task was the same for both the “Freeze-Object” and the existing freeze interaction technique. However, the participants had to move the device to translate the virtual object in the “Freeze-Object” interaction method, while in the existing freeze technique they performed touch screen gestures for the translation task. In the Freeze-Object interaction mode, a virtual 3D lamp remained in its position with a live background, and in the existing freeze method, the participants had to freeze the AR scene and keep moving the virtual object by dragging it on the touch screen. To see the target on the screen, participants had to keep the camera view on both targets and freeze the AR scene. Figure 9 illustrates the translation task in both conditions.

Fig. 9
figure 9

a Experiment environment consisting of the target for translation. b Participants tracking the target image. c Participants moving the device to see virtual lamp on specific target in Freeze-Object condition. d Participants performing translation using existing freeze method by touch gesture

Full size image

In the rotation task (Fig. 10), the participants had to rotate a virtual 3D table lamp clockwise around the Y-axis at angles of 45, 180 and 330° from the initial position using one finger. The rotation was set up in one axis to avoid interaction complexities because when users perform a rotation using a touch screen interface, it will lead to a deformable transformation in several axes.

Fig. 10
figure 10

a Participants performing rotation using Freeze-Object interaction method. b Participants performing rotation in existing freeze technique

Full size image

In the scaling task, the participants had to change the size of the virtual 3D table lamp along all three axes together to 0.5 times smaller, and 1.5 and 2 times larger from the initial position. The pinch-to-zoom gesture was used for scaling the virtual object. The system had the same working principle as rotation. The virtual object was reset to its initial position after the completion of each subtask. Figure 11 illustrates the scaling task in both conditions.

Fig. 11
figure 11

a Participants performing scaling in Freeze-Object. b Participants performing scaling in existing freeze technique

Full size image

5.3 Result

This section presents the results of the second experimental task. The main aim of this study was to see how users perform in a real application scenario using the “Freeze-Object” and the existing freeze interaction method. As with the first evaluation, the user’s performance was measured based on the task completion time, and the user’s preference for the interaction technique was gauged through a usability questionnaire.

5.3.1 Performance analysis

The task performance was measured based on the time taken by the user to complete the task using both interaction techniques. A paired sample t test with alpha p = 0.05 was performed for analysing the task completion time and to determine the significant difference between both conditions. The paired sample t test results (Table 4) showed that there was a significant difference between the translation tasks in both conditions. In the Freeze-Object (Translation1, M = 7.7337, SD = 3.96603) and the existing freeze technique (Translation2, M = 13.4025, SD = 4.22200); t (7) = −9.858, p = 0.00. According to the results of the translation task completion time, the users performed the translation task in the Freeze-Object significantly faster than the existing freeze interaction technique. There were no significant differences in the rotation and scaling tasks. However, for the overall task completion time, the Freeze-Object was faster than the existing freeze technique. Figure 12 shows the average task completion time for each of the three types of tasks for both conditions, as well as the overall average time for all the tasks.

Table 4 Paired sample t test result for second experiment
Full size table
Fig. 12
figure 12

Mean completion time of each and overall task

Full size image

5.3.2 Preference analysis

The users’ preference for the interaction technique was collected using a usability questionnaire (Table 3). The participants answered six usability questions after performing each manipulation task in both techniques. A Wilcoxon Signed-Rank Test was performed for the significant difference between the two conditions using Z statistics and p < 0.05. The results of the questionnaire showed that there was a significant difference in the translation manipulation task in terms of (Q2) easy to use (Z = −2.121, p = 0.034 [p < 0.05]) and engaging (Z = −2.000, p = 0.046, [p < 0.05]). According to results, the users felt that the translation task with the Freeze-Object technique was easier to learn and more engaging than the existing freeze interaction technique. No other significant difference was found (p > 0.05 for all responses). Table 5 shows the detailed results of the users’ preference.

Table 5 Wilcoxon sign rank test result for user’s preferences
Full size table

At the end of the whole study, the participants were asked to rank the interaction technique for each manipulation if they were required to do a similar task in the future, and to give their reason for choosing that particular interaction technique. For the translation task, 7 participants chose the Freeze-Object and 1 participant chose the existing freeze interaction technique. Seven participants mentioned that it was easy to move the device without depending on tracking, and that it was faster than touch gestures. For the rotation task, 6 participants chose the existing freeze method and 2 participants chose the Freeze-Object interaction technique. Six participants thought it was hard for them to hold the device and do the rotation, while 2 participants thought that rotation was easy in the Freeze-Object mode because they could adjust the virtual object according to the real world. For the scaling task, 5 participants chose the Freeze-Object mode and 3 participants chose the existing freeze technique. Five participants thought scaling in the Freeze-Object mode was good for adjusting the size of a virtual object in a real background, and 3 participants thought scaling in the existing freeze technique was more comfortable because they could put the device on the table. For the overall task, 5 participants chose the Freeze-Object and 3 participants chose the existing freeze technique. Five participants preferred the Freeze-Object because it was very easy, faster, more engaging and offered a real-time experience compared to the existing freeze technique. The user’s preference for the interaction technique can be seen in Fig. 13.

Fig. 13
figure 13

User’s preference on interaction technique

Full size image

6 Discussion

Although both interfaces supported the touch screen gesture for interaction, the main difference was that the Freeze-Object interaction allowed users to see an updated real-world view while the existing freeze technique showed a static real-world view as a still picture. The performance of the participants in each manipulation task, together with the feedback and observations from both user studies showed that there was a significant difference in the translation task. In the Freeze-Object interaction technique, the user did not have to hold the device with one hand and perform the touch gestures with the other hand for a moving object. After freezing the object, they had to move the device in a particular direction to where they wanted to see the object, which was faster. There were no significant differences for the rotation and scaling tasks. However, for the overall task, the users chose the Freeze-Object interaction technique.

There are a few reasons that could explain why the users chose the Freeze-Object interaction technique over the existing freeze method. First of all, most of the participants used the touch screen handheld device frequently and they were used to holding it with one hand and touching the screen with other hand. During the experiments, the participants could keep the device on the table after freezing the AR scene using the existing freeze technique. The existing freeze method was developed to provide users with a comfortable interaction, where they did not need to hold the device constantly to track the AR marker. Once they froze the AR scene, they could keep the device in any position and work comfortably. However, the participants felt more comfortable holding the device in their hand. Another reason was the real-time updated view. The Freeze-Object interaction technique allowed the users to see the live real-world view with a frozen virtual object, while the existing freeze method froze the whole AR scene, which gave the users less real-time AR experience. The real-time background also influenced the task time. The users’ task performance (task completion time) results showed that the users performed the translation task faster and easily using the Freeze-Object rather than the existing freeze method because with the Freeze-Object, the user just needed to move the device in the specific direction where they needed to see the virtual object, while in the existing freeze method, they needed to unfreeze the AR scene and move the marker to a new background before freezing the AR scene again, all of which required more time.

7 Conclusion and future recommendations

In this study, the freeze interaction method for handheld AR touch interaction was investigated, and an improved freeze technique, known as the ‘Freeze-Object’ interaction technique, was proposed. The Freeze-Object interaction technique allows a user to interact with a frozen virtual object with a real-world updated view in an AR environment. A description on the implementation of the Freeze-Object interaction technique has been given and the results of two user studies have been presented. Both user studies showed that users preferred the improved freeze technique over the existing freeze technique as they felt that the Freeze-Object interaction technique was easier for translation tasks and it provided them with a real-time experience.

In future, the focus will be on improving the proposed technique to provide more robust interaction to the users, while exploring the applications field where the Freeze-Object interaction technique is preferred. Currently, the Freeze-Object interaction technique supports three basic manipulation tasks (translation, rotation and scaling). In future, more object manipulation methods with the Freeze-Object technique will be explored, such as sensor-based manipulation. This will reduce the touch interaction for manipulating the virtual object. In future, different application areas will be explored where users need to deal with the real-world background in an AR scene. The proposed technique can be applied in different areas such as interior design, maintenance, etc.