1 Introduction

In contrast to most head-mounted displays (HMDs), which present only a computer-generated image, a video see-through HMD (VSHMD) allows users to perceive real-time video images from digital cameras mounted on glasses while also providing virtual information. VSHMDs are considered to be promising for many applications related to augmented reality and telepresence. For example, soldiers can train for combat in a real environment replete with three-dimensional (3D) audio and visual changes (Wurpts 2000). They must shift their viewpoint outside their eye position above walls or protective structures, while perceiving spatial information on a macroscopic viewpoint display through zoom cameras (Hughes et al. 2005). In architecture, VSHMD technology enables builders and architects to preview building structures and construction plans superimposed on actual sites, which is useful for comparing side wall materials and computing overall sizes (Bae et al. 2013). One particularly beneficial use of HMDs is that they allow rescuers to control robots or manipulators to perform life-saving tasks in hazardous environments such as burning buildings and sewage systems (Pretlove 1998). Furthermore, astronauts can view and collect materials from space via robot (Chintamani et al. 2008). To manually operate a robot in a remote environment, a sufficient 3D visual field is required to successfully process the characteristics of the remote space and to effectively accomplish tasks. However, the technical properties of VSHMDs induce sensory conflict, because visual displacement in terms of viewpoint mis-matching is caused by the difference between the camera position and the user’s eyes (Rolland et al. 1994). This sensory conflict induces visual and proprioceptive coordination performance errors.

To minimize side effects, various visual displacement problems induced by camera position have been evaluated in several studies; hence, guidelines have been suggested based on the adaptation process. In the first study on this topic, Rolland et al. (1995) reported that the visual displacement created by a camera position of 165-mm Z-axis and 62-mm Y-axis affects performance of hand–eye coordination tasks. In that study, it was found that the reaction time increased by 43% relative to the baseline, while pointing errors increased significantly in y and z spatial dimension tasks. However, in five trials using a VSHMD, the users quickly adapted to the discrepancy between the visual input information and the output behavior of the motor-induced visual displacement. Notably, Rolland et al. (1995) expressed concern regarding the aftereffects of visual displacement and argued the need for safety guidelines. Notably, Park et al. (2008) have described the visual displacement effects of 15 different camera positions by tracking the effects of Y- and Z-axis displacement on hand–eye coordination performance compared with the baseline for line tracing; dot placement; X, Y, Z spatial dimension tracing; and wingnut screwing. They found that performance at the 35-mm Y-axis was significantly better than that at the 70-mm Y-axis and the 130-mm Z-axis. Hence, they concluded that users can perform various visuomotor tasks without disturbance effects at displacements of no more than 35 mm on the Y-axis. In a study by Lee et al. (2013), 16 visual displacement combinations [55-mm X-axis, 55-mm Y-axis, 125-mm Z-axis, and 140-mm inter ocular distance (IOD)] were investigated. After the adaptation, no performance deterioration was observed across the conditions for pointing accuracy and foot placement tasks.

In the abovementioned previous studies, the influence on visuomotor performance of visual displacement due to the camera position on the X, Y, and Z axes was investigated. However, user adaptation to camera positions associated with broad visual displacement has yet to be widely explored. Therefore, in this study, the visual displacement is extended excessively to 300 mm and the eye–hand and eye–foot visuomotor coordination is investigated to explore the adaptation trend.

2 Background

2.1 Health and safety issues, sensory rearrangement, and dual adaptation

Various factors associated with the virtual environment and augmented reality produced by a HMD, including the field of view (FOV), resolution, and delay, can affect visuomotor performance. These issues have emerged in previous studies considering the capability and safety implications of HMDs, as well as other ramifications. According to Cobb (1999; Kennedy et al. 1994, 1995; Hakkinen et al. 2002), users have experienced symptoms such as postural instability that can affect the task performance. In keeping with those studies, Arthur (2000) has suggested that the FOVs of most HMDs restrict users’ normal FOVs and decrease task performance. Further, Kollenberg et al. (2010) have reported that the narrow FOVs of VSHMDs deteriorate search performance and restrict eye rotation, inducing a greater number of head movements. Furthermore, Moss et al. (2011) have shown that the VSHMD delay causes nausea, oculomotor discomfort, and disorientation, because of the mismatch between the visual and vestibular systems.

Although the user’s brain coordinates with their senses to compute spatial information, such as path trajectories, distance, speed, and accuracy, the various senses are not coordinated in an immersive virtual environment and augmented reality. Therefore, users should become accustomed to the virtual environment and adapt to the sensory rearrangement to reduce symptoms similar to those of carsickness and seasickness (Reason 1978). It is accepted that symptoms gradually decrease in 95% of users, but persist in 5% of users regardless of exposure (Reason 1974). In addition, post-adaptation aftereffects should also be considered (Stanney and Salvendy 1998; Stanney et al. 1999; LaViola 2000). That is, users adapt to the virtual environment and then ultimately return to the real world. This dual adaptation process generates aftereffects on the body coordination and proprioception that can persist for 24 h (McGonigle and Flook 1978).

2.2 Prism adaptation paradigm

Since Helmholtz (1909) first described prism adaptation in the late nineteenth century, this paradigm has attracted continuous studies. Generally, the prism adaptation process comprises pre-exposure, prism exposure, and post-exposure. Under pre-exposure conditions, the basic performance exhibited by participants using their natural (bare) eyesight is measured. During the prism exposure stage, the participants wear prism goggles, which laterally or vertically displace their visual fields, and undergo trials in which errors are thought to result from miscalibration between their vision and proprioceptive motor actions. However, as the trials are repeated, researchers observe reduced error rates and increased visual target adaptation. After the prism goggles are removed (the post-exposure stage), the participants exhibit error performance in the direction opposite that of the adaptation condition. This so-called negative aftereffect, indicating the persistent effect of adaptation, does not persist over long periods. Through real-world recalibration, participants recover their pre-exposure performance levels. This prism adaptation primarily comprises two mechanisms of spatial movement: recalibration and realignment (Redding and Wallace 1990). Recalibration occurs quickly to reduce performance errors, whereas realignment occurs slowly to produce long-term accuracy by coordinating input from various senses.

Based on the prism adaptation mechanism, visuomotor adaptation using prism goggles has been investigated in several studies. Frassinetti et al. (2002) investigated the possible correlation between the adaptation effect and the after-effect, which implied a possibility for long-lasting treatment for neglect patients using prism adaptation. Serino et al. (2007) similarly showed stable improvement of visuomotor abilities in neglect patients. Lang and Bastian (1999, 2001) have compared the performance of normal participants and patients with cerebellar dysfunction during the activity of catching a ball, which task is one of the typical measuring methods of visuomotor performance. Similarly, Alexander et al. (2011) have observed adaptation and aftereffects for generalized locomotion tasks. In line with those studies, Martin et al. (1996) have reported adaptation effects for throwing. The results of those studies suggest that the adaptation process emerges for many types of motor tasks.

3 Experiment 1: adaptation to excessive visual displacement

In this study, two experiments were conducted. In Experiment 1, the prism adaptation paradigm was used to assess and compare task performance under various visual displacement conditions. In Experiment 2, the same procedures were implemented as in Experiment 1, but for 3 consecutive days. Hence, the relatively long-term adaptation trend was assessed.

In Experiment 1, we studied visual displacements of 50, 100, 150, 200, and 300 mm on the X-axis, as shown in Fig. 1. We chose the X-axis shifts with reference to the values employed in the Redding and Wallace (1990) prism adaptation study, because there are few previous studies in which visual displacement on the X-axis of a VSHMD was examined. We applied the prism adaptation paradigm in this experiment (Savin and Morton 2008).

Fig. 1
figure 1

Camera horizontal shift relative to user’s natural eyes in VSHMD (VD visual displacement)

Full size image

3.1 Participants

Fifty-seven subjects were recruited through a board on a recruiting website, all of whom voluntarily agreed to participate. The participants were divided into 5 classes to evaluate their adaptation to various degrees of visual displacement as shown in Table 1. The classes were 6 males, 4 females, age: 25.9 (2.23); 6 males, 4 females, age: 25.5 (3.10); 6 males, 6 females, age: 28.7 (7.12); 5 males, 6 females, age: 24.4 (3.53); 4 males, 10 females, age: 26.7 (4.33); and each participant group is subjected to visual displacements of 50, 100, 150, 200, and 300 mm, respectively.

Table 1 Assigning participants to experimental conditions
Full size table

All participants were right-handed except for four persons, and all participants performed tasks with their dominant hand. The participants had normal or corrected-to-normal vision and reported no history of balance dysfunction. All participants signed a consent form prior to the experiment, which was conducted in accordance with the Helsinki Declaration and received approval from the Korea Institute of Science and Technology (KIST) Institutional Review Board (IRB).

3.2 Test setup

3.2.1 VSHMD

We used a prototypical VSHMD having two screens positioned 30 mm in front of the eye with a 30-frame per second (fps) refresh rate. The VSHMD had a straightforward install that did not require toe-in setup and did not generate vertical parallaxes. We set the 65 mm of IOD (Inter Ocular Distance) according to the study of Lee et al. (2013), which revealed that small disparities in IOD can be rapidly assimilated by human adaptation. In our experiment 65 mm of IOD was accepted by all subjects and caused no difficulty in performance of the assigned task, even though the participants’ interpupillary distances were distributed in the range of 55 to 70 mm. Our VSHMD provides 640 × 480 display resolution and horizontal and vertical FOVs of 48° and 18°. During the test, the participants were requested to hold a VSHMD in one hand and to perform tasks with their other (dominant) hand.

3.2.2 Ball delivery task setup for eye–hand visuomotor performance measurement

The participants were given a task of catching a 60-mm ball rolling down a board with a 25° slope while they were standing (Fig. 2). The bottom of the slope was at a height of 1060 mm from the floor. The start positions of the rolling balls were changed in each instance to provide a control against the effects of learning. When a computerized voice instructed the participant to catch the ball, the experimenter dropped the ball from the top of the slope. The participants were required to catch the ball using their dominant hand. When the participant failed to catch the ball, the experimenter placed another ball in his or her hand for the next step of the experiment. The experimenter scored the trial with a grade of 0 (failed to catch the ball), 1 (the ball hit the hand), or 2 (successfully caught the ball).

Fig. 2
figure 2

Ball catching setup used in experiment

Full size image

3.2.3 Footprint-stepping task for eye–foot visuomotor performance measurement

Soon afterward, the participants were requested to step forward to place both feet on a footprint target positioned on the ground (Fig. 3). The target was located on the ground 500 mm ahead of the participant’s position, and the participant had 2 s to complete the task. The experimenter recorded a score of 0 (stepped 40 mm or more outside the target), 1 (foot placed 40 mm or closer to the footprint), or 2 (stepped inside the footprint).

Fig. 3
figure 3

Foot placement device used in experiment

Full size image

3.2.4 Ball-throwing task for measuring difficult eye–hand visuomotor performance

Soon afterward, the participants were requested to throw balls into one of three 250-mm-diameter cylinders placed on the ground (Fig. 4). The target cylinder was placed 1 m in front of the foot placement target, and the three cylinders were 800 mm apart. For each trial, the experimenter designated a target cylinder and assigned a score of 0 (missed completely), 1 (the ball bounced off a cylinder), or 2 (the participant threw the ball into any cylinder).

Fig. 4
figure 4

Ball-throwing setup used in experiment

Full size image

3.2.5 Manual task

In the present study, the participants performed multiple motor tasks in the VSHMD environments. In detail, the participants performed the task in which they caught a rolling ball within 2 s and then immediately performed the footprint-stepping task within 2 s, before performing the task involving ball throwing into a cylinder within 2 s. After performing the ball-throwing task, they returned to their initial position within 2 s. Thus, one cycle comprised three tasks and attainment of the preparation position for the next trial. A session was defined as a sequence of 38 cycles, and 3 sessions of 114 cycles in total constituted one phase, with a rest period of 5 min between each session.

3.3 Procedure

The experimenter explained the procedure to all participants, who were then asked to engage in sequential trials according to the instructions given by the computer-generated voice. The voice automatically gave cues such as “now catch the ball,” “now step up to the target,” “now throw the ball to the first cylinder,” and “now return to the initial position.” A cue was presented every 2 s. In the baseline phase, the participants performed 8 cycles without the VSHMD (No-VSHMD phase). Then, the participants performed the tasks for 3 sessions while wearing the VSHMD (VSHMD phase). This phase was divided into 3 sessions of 38 cycles at 5-min intervals. After this phase, the participants entered the After-VSHMD phase, in which they removed the VSHMD and repeated 8 cycles under bare eye conditions. Figure 5 shows the procedure sequence.

Fig. 5
figure 5

Experimental procedure

Full size image

3.4 Results

As shown in Fig. 6, all VD (Visual Displacement) conditions significantly degraded the task performance (correct response score) compared to the baseline. As the VD was bigger, the performance degradation was higher. Also, each VD condition significantly degraded the performance compared to the 0 mm conditions except 50 mm condition, which means 50 mm of VD could be accepted by human perception in a short period of time—16 min in this experiment.

Fig. 6
figure 6

Task performance degradation for each VD

Full size image

Also, as shown in Fig. 7, the statistical analysis revealed no significant difference in aftereffect between the baseline and 50 mm condition. However, from the 100 to the 300 mm condition, a significant aftereffect emerged. Especially in the 200 mm and 300 mm conditions, significant aftereffect emerged compared to the 0 mm condition. Big amount of VD caused degradation in the task performance when they put off the HMD and return to the bear eye condition.

Fig. 7
figure 7

E task performance degradation of aftereffect for each VD

Full size image

A repeated measure analysis of variance (ANOVA) was performed for each VD condition (50, 100, 150, 200, and 300 mm). Figure 8 shows overall trend of task performance (correct response score). In early stage of the experiment, significant differences were observed between the baseline and VSHMD phases (p < .01), as well as After-VSHMD phases (p < .01) for all VD conditions. Finally, significant differences were sustained for the 100-, 150-, 200-, and 300-mm displacement conditions (p < .05). During the experiment, the correct response scores increased continuously in all VD conditions, providing evidence of temporal adaptation. Each session was analyzed throughout all VD condition using a 3 × 8 (session × cycle) repeated measures ANOVA.

Fig. 8
figure 8

Average correct responses for 3 tasks under 50-, 100-, 150-, 200-, and 300-mm displacement conditions during No-VSHMD, VSHMD, and After-VSHMD phases. Each point is 1 cycle

Full size image

Contrast analysis among sessions revealed significant differences between first and second session, as well as between first and third session, for all displacement conditions (p < .05). Significant differences were not observed between second and third sessions, except for the 300-mm displacement condition. The VSHMD and After-VSHMD phases were compared across the VD conditions to examine the differences in adaptation and aftereffects according to the VD. A 5 × 8 (displacement condition × cycle) mixed repeated measure ANOVA for the After-VSHMD phase revealed a main effect of displacement condition [F(4, 52) = 5.73, p = .001, mean square error (MSE) = .10, partial η2 = .31] and an interaction effect between displacement condition and cycle [F(28, 364) = 1.58, p = .031, MSE = .12, partial η2 = .11]. Multiple comparisons with Bonferroni adjustment showed differences between the 50-mm displacement condition and the 100- and 300-mm displacement conditions across 8 cycles (p < .05). Similar to the VSHMD phase analysis, a 5 × 8 (displacement condition × cycle) mixed repeated measure ANOVA for the After-VSHMD phase revealed a main effect of displacement condition [F(4, 52) = 5.12, p = .001, MSE = .30, partial η2 = .28] and an interaction effect between displacement condition and cycle [F(28, 364) = 1.67, p = .020, MSE = .07, partial η2 = .11]. Multiple comparisons with Bonferroni adjustment showed differences between the 50-mm displacement condition and the 100- and 300-mm displacement conditions at cycle 1 (p < .05).

Greater displacement on the X-axis yielded greater performance degradation in both the VSHMD and After-VSHMD phases. The performance level was significantly better under the 50- and 100-mm displacement conditions compared to the 300-mm case. These results are listed in Table 2.

Table 2 Results of adaptation and aftereffect analyses
Full size table

4 Experiment 2: adaptation improvement over 3 days

Kim et al. (2014) previously reported a negative effect of visual displacement on simulator sickness and also showed an adaptation effect over the course of 3 days. Experiment 2 of this study was conducted to assess the extent to which visuomotor adaptation and aftereffects improved under the 200- and 300-mm displacement conditions over 3 days. In this experiment, the same procedure as for Experiment 1 was implemented each day, for 3 consecutive days. The same 25 participants subjected to the 200-mm displacement condition [5 males, 6 females, age: 24.4 (3.53)] and the 300-mm displacement condition [4 males, 10 females, age: 26.7 (4.33)] of Experiment 1 participated in Experiment 2. Of these participants, 23 were right-handed and 2 were left-handed. One participant subjected to the 200-mm displacement condition experienced a health problem on the third day and, consequently, did not participate on that day. The experiment was conducted at the same place on each day.

4.1 Statistical analysis

Data from Experiment 2 were analyzed in the same manner as those from Experiment 1. A two-way repeated measures ANOVA was used to compare phase and session (in terms of cycle) to confirm the adaptation and aftereffects over the course of the 3 days. Then, a three-way mixed repeated measures ANOVA was conducted to compare adaptation and aftereffect improvement. The dependent measure was the average value of the visuomotor performance during the phase. Bonferroni post hoc analysis for multiple comparisons and contrast analysis were used.

4.2 Results

Consistent with the results of Experiment 1, adaptation and aftereffects occurred over the course of the 3 days for both displacement conditions, as shown in Fig. 9, where each point represents a cycle. Note that one VSHMD phase comprises 3 sessions including 5-min rest time between sessions.

Fig. 9
figure 9

Average correct performance responses for 3 tasks during No-VSHMD, VSHMD, and After-VSHMD phases for 200- and 300-mm displacement conditions collected over 3 days

Full size image

A 6 × 8 (phase × cycle) repeated measures ANOVA revealed significant differences between the No-VSHMD and VSHMD (3 sessions) phases, as well as between the VSHMD (3 sessions) and After-VSHMD phases (p < .05) for both displacement conditions over the course of the 3 days. The same analysis revealed significant differences between the VSHMD (3 sessions) and After-VSHMD phases on the second day for both the 200- and 300-mm displacement conditions (p < .05). However, there was no significant difference on the third day for either displacement condition. Furthermore, to evaluate the performance level during the VSHMD phase in detail, a 3 × 8 (session × cycle) repeated measures ANOVA was conducted. The results revealed significant differences between first and second sessions (p < .05), as well as between first and third sessions (p < .05). However, the same analysis revealed no difference between second and third sessions over the course of the 3 days, except for the first day and the 300-mm displacement condition (p < .05).

Next, to analyze the improvements over 3 days for both displacement conditions, a 3 × 2 × 8 (day × displacement condition × cycle) mixed repeated measures ANOVA for the VSHMD phase was performed. The results revealed a significant main effect of day [F(2, 44) = 43.43, p = .000, MSE = .38, partial η2 = .66] and a main effect of cycle [F(7, 154) = 2.84, p = .008, MSE = .12, partial η2 = .11]. An interaction effect of day and displacement condition [F(2, 44) = 3.45, p = .000, MSE = .38, partial η2 = .29] also revealed a significant difference between the first, second, and third days for both displacement conditions (p < .05), as well as a difference between the 200- and 300-mm displacement conditions for all 3 days (p < .05).

A 3 × 2 × 8 (day × displacement condition × cycle) mixed repeated measures ANOVA for the After-VSHMD phase revealed a significant main effect of day [F(2, 44) = 30.95, p = .000, MSE = .12, partial η2 = .59] and a significant interaction effect between day and cycle [F(14, 308) = 8.86, p = .000, MSE = .06, partial η2 = .29]. An interaction between day and cycle in the comparison analysis showed that there was a significantly different performance magnitude on each day within 4 cycles (p < .05). These results are summarized in Table 3.

Table 3 Adaptation and day effects over 3 days for 200- and 300-mm displacement conditions
Full size table

5 General discussion

The results of this study show that the participants’ visuomotor performance began to seriously degrade from an X-axis displacement of 200 mm upward, which is consistent with the results of a previous study (Lee et al. 2013). However, human adaptation is known to overcome excessive visuomotor difficulties to a great extent, as shown in previous adaptation studies (Harris 1963; Hay and Pick 1966; Luaute et al. 2009). In this work, we reaffirmed that the magnitude of the adaptation decreased as the visual displacement increased (Fernandez-Ruiz and Diaz 1999; Fernandez-Ruiz et al. 2000, 2011).

And the results showed the significant performance improvement across the displacement conditions over 3 days. However, for 200- and 300-mm displacement conditions, the performance increased gradually. The improvement growth was lower for the 300-mm displacement condition than for the 200-mm case, which means that considerably more time is needed for adaptation as the displacement increases. The participants may have been slower to adapt because this was difficult. So the excessive visual displacement may require a cognitive contribution to the adaptation process to achieve a degree of task performance.

There was no difference in performance magnitude in the After-VSHMD phase for the 200- and 300-mm displacement conditions. Further, performance for both displacement conditions did not fully recover to the No-VSHMD performance level despite performance improvement over the 3 days. Therefore, users who experience excessive displacement should exercise caution for a period of time after both wearing and removing the VSHMD and should expect decreased performance during those times. This outcome could also be attributed to such functional drawbacks as the poor display resolution, time delay, and FOV of the VSHMD. However, the performance level differed for the various visual displacements even though they were all subject to the same VSHMD disadvantages. A higher level of visual displacement appears to amplify the negative effects of those disadvantages.

6 Conclusion

This study assessed the capacity of human adaptation to visual displacement of up to 300 mm using a prototypical VSHMD designed by the authors. Based on the prism adaptation paradigm, 3 phases (No-VSHMD, VSHMD, After-VSHMD) were implemented in sequence, comprising catching, stepping, and throwing tasks. The results indicate that larger visual displacement induces a greater decrease in adaptation magnitude. In addition, adaptation improvements over 3 days were observed for visual displacements of 200 and 300 mm. With increased visual displacement, a significant amount of time and trial is necessary for adaptation. This is because the adaptation growth is decelerated by excessive displacement.

As a result of scientific and technological developments, VSHMD applicability ranges from daily life to telepresence and tele-robotics. However, certain hardware and software problems must be overcome, including limitations due to sensory conflict arising from the camera position or the virtual environment configuration. Excessive visual displacement can induce poor visuomotor task performance in the tele-operation. In some cases, the visual displacement could be quite large since the body structure of the tele-robot is different from that of the operator. This study implies that the operator can adapt this difference over time. However, when the performance level becomes close to that achieved under normal, or bare eye conditions, the operator should exercise caution, as a relatively long period of time is required to achieve the level of performance attained under normal conditions. We suggest that the results herein be made available to researchers, designers, and developers in relevant application areas.