Abstract
Perturbations in virtual reality (VR) lead to sensorimotor adaptation during exposure, but also to aftereffects once the perturbation is no longer present. An experiment was conducted to investigate the impact of different task instructions and body representation on the magnitude and the persistence of these aftereffects. Participants completed the paradigm of sensorimotor adaptation in VR. They were assigned to one of three groups: control group, misinformation group or arrow group. The misinformation group and the arrow group were each compared to the control group to examine the effects of instruction and body representation. The misinformation group was given the incorrect instruction that in addition to the perturbation, a random error component was also built into the movement. The arrow group was presented a virtual arrow instead of a virtual hand. It was hypothesised that both would lead to a lower magnitude and persistence of the aftereffect because the object identity between hand and virtual representation would be reduced, and errors would be more strongly attributed to external causes. Misinformation led to lower persistence, while the arrow group showed no significant differences compared to the control group. The results suggest that information about the accuracy of the VR system can influence the aftereffects, which should be considered when developing VR instructions. No effects of body representation were found. One possible explanation is that the manipulated difference between abstract and realistic body representation was too small in terms of object identity.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
1 Sensorimotor adaptation in VR
Through learning processes, movement and perception are associated in such a way that people can precisely manipulate their environment. Disturbances of these learned associations can be caused by changes in our body or in the environment and are called perturbations. Sensorimotor adaptation is the gradual adjustment of our motor commands to compensate for such perturbations (Della-Maggiore et al. 2015). In the past, researchers have investigated sensorimotor adaptation primarily by using lenses, prisms or mirrors that alter the visual field by displacing, inverting, tilting or otherwise distorting it (Welch and Warren 1980).
The first study to systematically investigate sensorimotor adaptation in virtual reality (VR) was published by Biocca and Rolland (1998) and Rolland et al. (1995). The rationale for the research was the practical problem that with video-based head-mounted displays (HMDs) the virtual eye position did not correspond to the actual eye position because the camera could not be placed where the person's eyes were already located. The video-based HMD was compared to a dummy HMD that had the same weight, centre of mass and field of view, but no video transmission, to see if this technical problem influenced task performance. Participants in the video-based HMD condition showed movement errors at the beginning (= direct effect) but adapted to this visual displacement over time. When the participants took off the HMD, they also showed a change in their hand–eye coordination. These aftereffects appeared, for example, in the form of less accurate pointing movements compared to the baseline measurement (Biocca and Rolland 1998).
The study by Biocca and colleagues shows just one of many possibilities in which perturbations can arise in VR. First, reality can be misrepresented due to technical limitations, e.g. calibration errors. In addition, technological developments do not have the goal of VR as a perfect simulation of reality, because this will probably never be achieved. Instead, they have the goal of "perfecting" the user interface (Doerner et al. 2022). Therefore, some erroneous representations of reality are also intentional, such as in redirected walking (Nilsson et al. 2018). Here, the walking movement corresponds to a different walking movement in VR due to spatial restrictions. Another example of visuomotor illusions is reach redirection, which allows targets to be reached that are beyond actual reach while maintaining some semblance of body ownership (Cohn et al. 2020; Gonzalez et al. 2022). Furthermore, discrepancies between perception and movement are intentionally built in as part of the VR experience. These include, for example, body illusions in which body proportions are changed (Kilteni et al. 2012b).
Both body illusions and visuomotor illusions occur when there is still a sense of embodiment despite the discrepancies. A sense of embodiment emerges when the properties of the virtual body are processed as if they were the properties of one’s own biological body. The sense of embodiment consists of three subcomponents: the sense of self-location (= the feeling of being located in a determined volume in space), the sense of agency (= the sense of having global motor control), and the sense of body ownership (= one’s self-attribution of a body, Kilteni et al. 2012a). Factors that enhance these subcomponents, and thus the sense of embodiment, include the first-person perspective, sensory and visuomotor correlations within critical boundaries, and morphological similarity between the real and virtual body, among others (Kilteni et al. 2012a).
Once a sense of embodiment has been achieved, sensory impressions and movements are aligned with the virtual body representation. One example is the self-avatar follower effect: users tend to unconsciously follow their avatar when the virtual body does not overlay with their physical body (Gonzalez-Franco et al. 2020). Burin et al. (2019) found that deviations between intended and actual virtual movement led to stronger adjustment in favour of the virtual movement when the avatar was shown from a first-person perspective instead of a third-person perspective. A stronger sense of embodiment in VR thus seems to lead to stronger movement adjustment.
The sense of embodiment could also be important in the discussion of whether and to what extent sensorimotor adaptation differs between VR and reality. Studies show that visual displacement in VR leads to a similar adaptation (Wilf et al. 2021, 2023; Cho et al. 2022) and is influenced by similar factors (Gammeri et al. 2020) as in prism adaptation. Wähnert and Gerhards (2022), for example, found empirical evidence that the positive relationship between the number of interactions during perturbation and the aftereffect, which is already known from prism experiments (Dewar 1970; Welch 1971; Fernández-Ruiz and Díaz 1999), can be generalised to VR. However, comparative studies between VR and reality have also found differences. In VR, sensorimotor adaptation in a visuomotor task is associated with less implicit learning (Anglin et al. 2017) and more cognitive load (Juliano et al. 2022) than in a real environment. These differences may be due to how the body is perceived in VR and thus how movement errors are attributed.
2 Error attribution
The nervous system interprets sensory feedback with regard to its causes (Wei and Körding 2009). Hereafter, in the context of movement errors, the causal inference of sensory feedback is referred to as error attribution. In Welch's (1972) prism experiment, error attribution was manipulated using instructions. In contrast to the informed group, the misinformed group was given the misinformation that when they performed a pointing movement in the dark, they saw their own luminous finger, which was visually displaced by the prism glasses. In fact, the finger belonged to the experimenter and no actual visual displacement via prism took place. This misinformation was intended to enhance adaptation by suggesting that the displaced luminous dot represented the participant’s hand, and led to an aftereffect about twice as large as the true information given to the informed group that the experimenter moved the luminous dot. Following this study, Welch and Warren (1980) formulated the unity assumption. According to this assumption, an important precondition for the occurrence of a discrepancy between different sensory feedback is that the discrepant information comes from a single object, e.g. one's own hand. If the objects are different, there is no intersensory discrepancy and thus no reason for adaptation (Welch and Warren 1980). Bedford later incorporated this assumption and generalised it within the framework of his theory of object identity (Bedford 2001). Object identity is thus not only required for sensorimotor adaptation, but in general when information from different times, places, modalities, and eyes is to be integrated into a coherent view of the world.
Unity assumption and object identity can be influenced by various factors. Bedford (2001) refers to the distinction made by Radeau and Bertelson (1977), according to which the interpretation of single events is influenced by structural and cognitive factors. Structural factors are abstract properties of the sensory input, such as synchrony. Cognitive factors are “features of the situation which communicate knowledge about the distal situation from which the data originate, and so indicate whether a unitary interpretation is correct or not” (Radeau and Bertelson 1977, p. 137). Such features can consist of verbal instruction or a realistic context. Realistic context refers to “a context more or less evocative of situations known to produce correlated sense data in the two modalities” (Radeau and Bertelson 1977, p. 137). The aim of the present experiment was to investigate whether the two cognitive factors verbal instruction and realistic context influence sensorimotor adaptation in VR, and whether this relationship is mediated by error attribution.
Welch (1972) has already shown that instructions can have an influence on the aftereffect in prism adaptation. In real environments, it is usually assumed from previous experience that the felt hand and the seen hand belong to the same object. In VR, however, the user must be convinced, e.g. through visual-tactile or visual-motor synchrony, that the virtual body parts correspond with the real body parts in order to evoke a sense of embodiment (Slater et al. 2009). Therefore, in VR it should be possible to manipulate the perception of object identity by instructions without limiting visual information to a minimum, as it was necessary in Welch's experiment. Besides that, Welch used a misinformation about a correspondence between a luminous dot and one's own hand to increase adaptation and thus the aftereffect. We wanted to investigate whether misinformation about an erroneous correspondence between the virtual hand and the real hand would reduce the aftereffect. Thus, the following hypotheses were formulated:
H1: In VR, misinformation about an erroneous correspondence between a virtual and a real hand leads to a) a smaller and b) a less persistent aftereffect than no misinformation.
According to Radeau and Bertelson (1977), the second cognitive factor that influences the interpretation of single events is the realistic context. As already described in the last section, body representation seems to play an important role in the alignment with the virtual body. Therefore, we have limited the realistic context to the realistic body representation. Several prism studies have already been able to show that a more abstract representation of the body, e.g. a cursor, leads to a lower aftereffect (Clower and Boussaoud 2000; Norris et al. 2001; Veilleux and Proteau 2015; Aziz et al. 2020). One possible explanation for these results would be that more abstract representations increase the uncertainty that the abstract representation of the hand and the felt real hand belong to the same object, so that a discrepancy between them gives less cause for adaptation (Norris et al. 2001). In VR, body representations can be changed very easily (Kilteni et al. 2012b; Banakou et al. 2013). The present study investigated whether a more abstract body representation in VR affects sensorimotor adaptation. For the control group, the hand model from the experiment by Wähnert and Gerhards (2022) was adopted, with which large and persistent aftereffects were found. The abstract representation was intended to be achieved by displaying a 3D arrow instead of the virtual hand. The following hypothesis was formulated:
H2: In VR, the 3D arrow representation leads to a) a smaller and b) a less persistent aftereffect than the virtual hand representation.
If the perception of object identity is manipulated by instructions and body representation, then this should influence error attribution. More precisely formulated: If the real hand and the virtual representation are regarded as a unit or as one object, then errors related to this virtual representation should be assigned a different relevance. The relevance estimation model (Wei and Körding 2009) states that the nervous system estimates whether discrepancies between vision and proprioception are due to intrinsic factors that lie within the sensorimotor system or to external factors that are unrelated to the sensorimotor system, e.g. calibration errors. If visual cues are inferred to be irrelevant, they will be disregarded for movement production and thus adaptation (Wei and Körding 2009). True adaptation is characterised by generating new mappings of internal representations of one's own body. These are not restricted to the particular context in which perturbations were produced (Fleury et al. 2019). Applied to misinformation and abstract body representation, this would mean that both would lead to a lower aftereffect because the visual perturbation is seen as irrelevant to adaptation, i.e. attributed to a faulty system rather than to the self. The perception of error relevance was assessed by self-report. The following two hypotheses were formulated:
H3: Effect of misinformation on a) magnitude and b) persistence is mediated by error relevance.
H4: Effect of 3D arrow representation on a) magnitude and b) persistence is mediated by error relevance.
3 Methods
3.1 Participants
A total sample of 30 individuals were recruited to participate in this laboratory study. Two participants were excluded due to simulation sickness and task execution problems and were subsequently replaced, resulting in an analysis sample of 30 participants (aged 22–29 years; 12 men, 18 women). We adopted the sample size from the experiment by Wähnert and Gerhards (2022), who also analysed 10 individuals per group and found differences in terms of the aftereffect. Furthermore, according to our G*Power a priori power analysis for one-sided t-tests in a linear model regression, which we used to test the main hypotheses (see data analysis), a total sample size of 30 individuals is sufficient to detect effect sizes of \({f}^{2}=0.31\) or partial \({R}^{2}=0.23\) with \(1- \beta =0.9\). Eligibility criteria included healthy, right-handed individuals with normal or corrected-to-normal vision, native German language proficiency, no previous negative VR experience, a minimum body height of 1.55m, no physical impairment of the right arm, and no neurological disease affecting the upper body. All participants gave their informed consent prior to data collection and received course credits or money as compensation. The present study was approved by the Research Ethics Committee of the Department of Psychology and Ergonomics at the Technische Universität Berlin.
3.2 Apparatus
As shown in Fig. 1a, the participants sat at a table during the experiment with their heads resting on a chin rest. A cardboard marker was placed on the table to indicate the distance from the virtual wall. It served as an orientation to not put the arms too far forward after the pointing movement. The virtual environment (see Fig. 1b) was presented via an HMD with built-in off-ear headphones (Valve Corporation 2023). Acoustic signals were given via the headphones to indicate the start and end of the pointing movement. An HTC Vive tracker was attached to the participant's right hand to track the position of the index finger. A finger orthosis was also attached to stabilise the middle and index fingers. In the virtual environment, the participants sat at a table with a wooden board on which the targets appeared. The setup of the virtual environment was the same as in the experiment by Wähnert and Gerhards (2022).
3.3 Procedure and experimental design
After reading the participant information and agreeing to the informed consent form, participants were asked to complete a demographic questionnaire. Afterwards, the participants were informed about the VR task and procedure and had the opportunity to ask questions. After adjusting the HMD to the participant's pupil distance, attaching and calibrating the Vive tracker, placing the participant's head on the chin rest, and putting on the HMD, the VR experiment began.
Before the pointing task started, participants were instructed to move their right hand along the front edge of the table from the centre to the right corner. The correspondence between visual and haptic feedback was intended to reinforce the perception of object identity between the real hand and its virtual representation.
The pointing task was the same as in Wähnert and Gerhards (2022). The position of the target could deviate both vertically and horizontally by 3 cm from the centre position, resulting in nine possible target positions. The sequence of the target positions was predetermined. The position and sequence of the targets can be found in the description of the data set under the link https://doi.org/10.14279/depositonce-17735. The participants were instructed to carry out the pointing movements as accurate as possible. Movement speed was not instructed but measured and tested for group differences. The experimental procedure was based on the original prism adaptation paradigm (Kornheiser 1976), and the terminology for the different paradigm phases was adopted from Prablanc et al. (2020). The familiarisation phase consisted of 30 trials: 10 with concurrent feedback of the pointing movement and 20 with only terminal feedback of the pointing movement, as in all subsequent phases. After recording the baseline performance in the 10 trials of the baseline phase, the virtual environment was visually displaced to the right by 11.31 degrees (or 20 dioptres) in the exposure phase, which consisted of 35 trials. It is known from prism research that both the size of the deviation and the number of trials in the exposure phase have an influence on the aftereffect (Dewar 1970; Welch 1971; Fernández-Ruiz and Díaz 1999). The parameters set (20 dioptres and 35 trials) are based on the experiment by Wähnert and Gerhards (2022), who were able to demonstrate a strong and persistent aftereffect in a virtual environment. In the de-exposure phase, which consisted of another 30 trials, the visual displacement was removed. Participants were informed about both the visual displacement and its removal by instructions in the virtual environment before the corresponding phase. Performance in the de-exposure phase was used to assess the aftereffect. To measure its magnitude, a target position was presented in the first trial of the de-exposure phase that had not been used in the previous trials (see data analysis). During the entire VR experiment, a ten-second rest period was inserted after a maximum of 15 trials. After the VR experiment, the participants answered questions on error relevance and realism perception. Finally, the participants were debriefed, and remaining questions were answered. The whole experiment was constructed to last 45 to 60 min. The VR part of the experiment lasted no longer than 20 min.
Participants were randomly assigned to one of three groups: the control group, the misinformation group, and the arrow group. The control group was equivalent to the group from the experiment by Wähnert and Gerhards (2022) with thirty-five pointing movements. The misinformation group differed from the control group with respect to the misinformation. Participants in the misinformation group were told that, during the exposure phase, a random error component was built into the movement in addition to the visual displacement of the environment. The difference between the random error component and the visual displacement was made clear in the instructions and when asked by the participants. The arrow group differed from the control group with respect to the representation of the hand. Instead of a virtual hand, a 3D arrow was displayed (see Fig. 1c). The tip of the arrow corresponded to the tip of the middle finger. The size and movement features were identical to those of the virtual hand.
3.4 Data analysis
Statistical analyses were conducted using R (version 4.2.2, R Core Team 2023). The data of the experiment can be found in a data repository under the link https://doi.org/10.14279/depositonce-17735. The hypotheses were tested by analysing the magnitude of the aftereffect, the persistence of the aftereffect and the error relevance. The post hoc power analysis was conducted with \(1-\beta\) = 0.9.
According to Prablanc et al. (2020), there are several ways to measure and calculate the magnitude and persistence of aftereffects. In this paper, we based our measurement methods on Fernández-Ruiz and Díaz (1999) and Wähnert and Gerhards (2022). First, we baseline-corrected the performance in the de-exposure phase for each participant. Magnitude was defined as the horizontal deviation in the first trial of the baseline-corrected de-exposure phase. There were two reasons why only the first trial was selected. First, in the first trial of the de-exposure phase, a target position was displayed that had not been used in the previous trials. The rationale for changing the target position for measuring the magnitude is that if there is a sensorimotor realignment beyond strategic or cognitive effects, this should generalise beyond the task conditions in which the visual displacement was induced (O’Shea et al. 2014). Generalisation to new target positions has already been shown for prisms (Bedford 1989) and virtual visual feedback (Vetter et al. 1999). Second, only the first trial was performed without feedback experience from the de-exposure phase. The remaining trials were used to measure persistence, which is the robustness of the horizontal deviation despite feedback. Persistence was defined as the mean of the absolute horizontal deviation in the remaining 29 trials of the baseline-corrected de-exposure phase. In addition to this approach of predetermining the number of trials and looking at the average performance within these trials, another possibility would be to set a certain performance level and analyse how many trials are required to reach and maintain this level. We decided against this option because we see a methodological problem here: participants who have reached the performance level in a trial do not remain constant at this level, but show fluctuations. It is difficult to determine what proportion of these fluctuations is already present in the baseline phase and can be neglected, and what proportion is due to persistence of the aftereffect. Even if a baseline range can be defined, it is still difficult to determine the persistence, e.g. in the case of single outliers after staying within the baseline range for several trials.
To analyse the group differences under hypotheses H1 and H2, we applied a linear model regression, dummy-coding the three groups with the control group as reference. The hypotheses about the differences of the misinformation and the arrow group compared to the control group were formulated as statistical hypotheses about the regression coefficients (Fox 2016).
Error relevance was measured at the end of the VR experience by asking the participants to estimate the reason for their movement error on two continuous 10 cm scales. One scale assessed the error attribution to oneself, the second scale assessed the error attribution to the system. Error relevance was defined as the difference between the two responses, with larger values indicating stronger self-attribution and thus greater error relevance.
In order to test the two mediator hypotheses H3 and H4, the conditions according to Baron and Kenny (1986) were analysed. Applied to hypotheses H3 (magnitude of aftereffect) and H4 (persistence of aftereffect), these can be formulated as follows: (a) there is an effect of the group on error relevance, (b) there is an effect of error relevance on the aftereffect, and finally (c) the effect of the group on the aftereffect becomes smaller when error relevance is added to the linear model (see Fig. 2).
Prior to hypothesis testing, we conducted a manipulation check. To test whether the arrow group perceived the hand representation more abstractly than the control group, we asked participants to rate the degree of realism of the pointer (realism perception: “How realistic would you rate the representation of the pointer in VR compared to the rest of the VR environment?”).
4 Results
4.1 Demographics and manipulation check
There were no significant differences between the groups for age (F(2,27) = 0.62, p = 0.54, \({\eta }^{2}\)< 0.32), gender (χ2(2) = 0.83, psim = 0.89, V < 0.65), VR experience (F(2,27) = 1.52, p = 0.24, \({\eta }^{2}\)< 0.32), tiredness (F(2,27) = 1.08, p = 0.35, \({\eta }^{2}\)< 0.32) and baseline performance (F(2,27) = 0.17, p = 0.85, \({\eta }^{2}\)< 0.32). No significant group differences were found for the time from start to end of the pointing movement in the baseline (F(2,27) = 0.39, p = 0.68, \({\eta }^{2}\)< 0.32), exposure (F(2,27) = 0.26, p = 0.78, \({\eta }^{2}\)< 0.32) and de-exposure phase (F(2,27) = 1.29, p = 0.29, \({\eta }^{2}\)< 0.32). As expected, the realism of the arrow in the arrow group was perceived significantly worse than that of the virtual hand in the control group (t(18) = 1.75, p = 0.049, \(\widehat{d}\) = 0.78).
4.2 Magnitude and persistence
Figure 3 shows the horizontal deviation depending on group and phase. Both the direct effect in the exposure phase and the aftereffect in the de-exposure phase are present in all groups.
The magnitude of the aftereffect, i.e. the baseline-corrected horizontal deviation in the first trial of the de-exposure phase, for the three groups is shown in Fig. 4 on the left side. The one-sided t-tests on the model coefficients revealed that both the misinformation group (t(27) = 1.38, p = 0.91, \({d}_{\psi }\)< 1.35Footnote 1) and the arrow group (t(27) = 0.17, p = 0.57, \({d}_{\psi }\)< 1.35) showed no significant differences from the control group. Thus, the magnitude of the aftereffect does not seem to be influenced either by the misinformation about an erroneous correspondence between virtual and real hand (H1a) or by the 3D arrow representation of one’s own hand (H2a).
The statistical model used to analyse the magnitude was fitted for the variable persistence, which is defined as the mean absolute baseline-corrected horizontal deviation in the de-exposure phase without the first trial. Figure 4 on the right side shows the results. There was a significant difference between the misinformation group and the control group (t(27) = -2.46, p = 0.01, \({\widehat{d}}_{\psi }\) = 1.10). This confirms the hypothesis that the misinformation about an erroneous correspondence between a virtual and a real hand leads to a less persistent aftereffect (H1b).
In contrast, the arrow group showed no significant differences from the control group (t(27) = 1.99, p = 0.97, \({d}_{\psi }\)< 1.35). The hypothesis that the 3D arrow representation leads to a less persistent aftereffect (H2b) can be rejected (a one-sided t test in the other direction would even have reached significance).
4.3 Error relevance
A linear model with error relevance as criterion and the groups as predictors was fitted. The proportion of variance explained was not significantly different from zero (F(3,26) = 0.75, p = 0.53, R2 = 0.08). Error relevance showed no significant correlation with magnitude (r = 0.05, p = 0.81) or persistence of the aftereffect (r = -0.12, p = 0.53). Adding the error relevance to the linear model for analysing the group effects did not result in any changes in the group differences. As the conditions for a mediator effect were not met (Baron and Kenny 1986), it cannot be confirmed that error relevance mediates the effect of misinformation (H3) or 3D arrow representation (H4) on magnitude and persistence.
4.4 Explorative data analysis
The groups were tested for differences in pointing movement during exposure. As in the de-exposure phase, a distinction was made between baseline-corrected performance in the first trial and mean absolute performance in the remaining trials. A one-way ANOVA was performed in each case. No significant group differences were found (F(2,27) = 1.49, p = 0.24, \({\eta }^{2}\)< 0.32 and F(2,27) = 0.88, p = 0.43, \({\eta }^{2}\)< 0.32).
The standard deviation of the realism perception was almost twice as large in the arrow group (SDArrow = 2.75) as in the other groups (SDControl = 1.81, SDMisinf. = 1.52), indicating large individual differences within the arrow group. Therefore, realism perception was added as a metric predictor with interaction in the linear model. Within the arrow group, the slope of the regression line was significant (t(24) = 1.71, p = 0.04, \({\widehat{d}}_{\psi }\) = 0.23). In other words: within the arrow group, a higher realism perception led to a higher magnitude (see Fig. 5). Thus, the hypothesis about the effect of the 3D arrow representation on magnitude (H2a) cannot be confirmed at the group level, but there is some evidence that the realism perception of the arrow had an influence on magnitude.
Figure 3 gives an indication that the horizontal deviations across the groups showed certain patterns. This is particularly evident in the baseline phase: across all groups, trial 1 and trial 9, for example, showed a high horizontal deviation to the left. These are the only trials in the baseline phase with a left target position (as the horizontal deviation is the dependent variable in the experiment, vertical differences in the target positions are not considered here). To investigate the effect of target position on horizontal deviation, a 3 (within; horizontal target position: left vs. centre vs. right) × 3 (within; phase: baseline vs. exposure vs. de-exposure) × 3 (between; group: control vs. misinformation vs. arrow) mixed ANOVA was conducted. The first trial in each of the exposure and de-exposure phases was removed from the analysis to prevent confounding effects from distorting the results. For these trials, confounding effects existed because they had the largest horizontal deviation and were assigned a specific target position at the same time. Figure 6 shows the descriptive results. The two main effects horizontal target position (F(2,54) = 56.44, p < 0.001, \({\widehat{\eta }}_{p}\)= 0.68) and phase (F(2,54) = 25.07, p < 0.001, \({\widehat{\eta }}_{p}\)= 0.48) reached significance. In addition, the interaction between horizontal target position and phase was significant (F(4,108) = 7.69, p < 0.001, \({\widehat{\eta }}_{p}\)= 0.22). There were no other significant effects.
5 Discussion
In the present experiment, the effect of error attribution through instruction and body representation was investigated. The hypotheses regarding instruction could only be partially confirmed: the misinformation group, which was instructed that a random error was built in in addition to the visual displacement, did not show a significantly different magnitude, but a significantly smaller persistence than the control group.
In terms of magnitude, Welch's (1972) results could not be replicated. Although the same underlying mechanisms were assumed, there are many differences between these two experiments in terms of the manipulation. First, Welch's misinformation created the illusion of object identity between the luminous finger and the participant's own finger, and to achieve this, the visual environment was reduced to a minimum (target and feedback were shown in darkness). In the present experiment, misinformation was used instead to break this illusion within an immersive VR environment. Perhaps the immersion of the VR was too high to break the illusion to such an extent that an effect on magnitude could be shown here. Second, Welch's misinformation manipulated the perceived reason for the displacement of the feedback. The informed group correctly believed that the luminous dot belonged to another person who was simply providing feedback offset from their own finger. The misinformed group falsely believed that the displacement was caused by prism glasses. In the present experiment, all participants were instructed that the virtual environment was visually displaced. The manipulation only affected the perceived accuracy of the feedback. Thus, the global displacement was not questioned, only the local feedback within the displacement. Although this may have reduced the effect of the instructions, it should also have led to a lower unity assumption, i.e. that the virtual hand is seen less as the same object as one's own hand. Finally, it should be noted that magnitude was only measured using one trial. To increase reliability, several pointing movements without visual feedback experience should be included next time.
In terms of persistence, the results were in accordance with the hypothesis. Besides the interpretation with respect to the unity assumption, further explanations are possible. For example, the false information of a random perturbation might have led to a higher perception of feedback variability. Studies have already shown that the actual additional trial-to-trial variability of a perturbation reduces the extent of adaptation during exposure (Havermann and Lappe 2010; Fernandes et al. 2012). According to Bayesian models, manipulating variability changes how much participants rely on new sensory information (Fernandes et al. 2012). Although the actual variability of the feedback was not manipulated, the illusion of higher variability may have led to greater use of cognitive strategies. When the contribution of cognitive strategies is increased, the aftereffects are reduced. Michel et al. (2007), for example, could show that a constant displacement, compared to a gradual displacement, led to greater awareness of the perturbation and was thus associated with a smaller aftereffect. Accordingly, the instruction that drew attention to a possible perturbation may also have increased awareness and thus led to less persistence of the aftereffect.
The hypotheses regarding body representation could not be confirmed, neither in terms of magnitude nor persistence. One possible explanation could be that the difference between hand and arrow in terms of how much it affects object identity was too small. All studies that were able to show an effect of an abstract body representation on the aftereffect compared different media: direct view of the hand vs. a video of the hand (Norris et al. 2001) vs. computer-generated image of a cursor or a line (Clower and Boussaoud 2000; Norris et al. 2001; Veilleux and Proteau 2015; Aziz et al. 2020). In the present experiment, abstraction was varied within the same medium, namely within VR. Comparisons between media include other differences that can influence aftereffects, such as resolution. The effects within a medium may be much smaller, as can be seen in our experiment, where only the representation is changed.
To reveal possible effects of representation within a medium, the difference between abstract and realistic representation should be further increased in terms of object identity. To further reduce object identity in an abstract representation, other forms of representation than an arrow could be investigated, which are perceived as less realistic. Realism perception was significantly associated with magnitude within the arrow group. Those who considered the arrow to be a realistic representation of their hand showed a higher magnitude of the aftereffect. In the research of body ownership, it has been shown that the viewed object should match a visual representation of the body part to elicit a sense of body ownership (see overview in Tsakiris 2010). For example, hand-shaped objects induce stronger body ownership illusions than non-hand-shaped objects (Haans et al. 2008). In the current experiment, the subjective realism rating of the arrow could to some extent reflect an identification with a computer-generated arrow as a valid representation of one's own hand, due to the habit of working on the computer. If this hypothesis is true, then the realism assessments should be related to the experience with computer-generated arrows in everyday life, which, however, was not measured in the context of this experiment. In future studies, in order to increase the group differences, other forms of representation can be investigated that are not already used in computer-influenced everyday life, but those that are not yet associated with hand and pointing movements.
In order to increase the object identity in the realistic representation, a better virtual hand model should be used, one that corresponds as closely as possible to the users' hands. To meet the individuality of the users, customisations can be made, for example to their sex and the colour and size of their hands. Another aspect concerns the realism of the movement. In the present experiment, the index and the middle finger were fixated to enable finger tracking. This fixation suits an arrow with fixed components better than a hand with movable fingers. This could be the reason for the surprising opposite effect that the arrow group showed a more persistent aftereffect than the control group. In future studies, a data glove, for example, could help to reduce this incompatibility and to transfer the mobility of the fingers into virtuality.
The hypotheses regarding error relevance could not be confirmed. Neither the misinformation group nor the arrow group showed an effect on perceived error relevance. Previous research has already revealed the influence of error relevance at the behavioural level (Wilke et al. 2013). The idea in this experiment was to measure the perception of error relevance through self-report. However, it is possible that the measurement of error relevance was not sensitive enough. The formulation referred to a general movement error and was thus too unspecific. Individual interpretations of what the movement error refers to probably increased the variance, so that no effect could be found. In addition to the error relevance, the indirect measurement of the sense of embodiment by questions about its subcomponents self-location, agency, and body ownership would have been helpful for the interpretation of the results (see an overview in Kilteni et al. 2012a).
In addition to the hypothesis testing, a further exploratory data analysis revealed an interesting effect: the target position led to a systematic movement error, namely to a stronger outward deviation, i.e. right positions led to a stronger deviation to the right and left positions to a stronger deviation to the left. This target position effect can probably be attributed to an underestimation of the egocentric distances in VR. Since no concurrent feedback about the movement was given, the movement planning had to be based on the target distance estimation. It is already known that distances are underestimated in VR (Renner et al. 2013). As can be seen in Fig. 7, the biased distance estimation may have led to biased movements as well. Due to the lack of concurrent feedback about the movement, this bias was not corrected during the movement and was reflected in a systematic movement error. This is relevant for the present study insofar as the magnitude was measured over a single target position, namely “down-right”. Since the aftereffect is defined as a deviation to the left in this experiment (because of the visual deviation to the right), the magnitude of the aftereffect was presumably reduced by the target position effect for all groups, which should be considered in the general evaluation of its size.
To our knowledge, there are no studies that focus on outward deviations in all phases without concurrent feedback. Redding and Wallace (2006) investigated the differentiation between recalibration and realignment, as well as untrained and trained target positions. They found that recalibration aftereffects were stronger when de-exposure and exposure conditions were similar regarding limb posture and training positions. Realignment aftereffects were not dependent on these conditions and generalised nonassociatively. It was also depicted that for terminal feedback in the de-exposure phase, targets on the left side had a stronger realignment aftereffect than targets positioned straight-ahead, and these again showed a stronger aftereffect than targets positioned on the right. The authors claim that proprioceptive targets to the right of the de-exposure straight-ahead position are trained, targets to the left are untrained (Redding and Wallace 2006). In our study, the left and right outward deviation was also found in the baseline phase with only ten trials, without concurrent feedback, and with targets positioned left, right, and straight-ahead. These results are related to, but not the same as, the outward biases, as we found a stronger outward deviation during all phases, i.e. baseline, exposure, and de-exposure phase.
Thomas (2012) investigated movement exercises crossing and not crossing the body midline via a sensorimotor adaptation task. The task differed from a prism adaptation task, as the movement was not explicitly hidden, i.e. the used hand was held besides the body and controlled a joystick, the targets were positioned equidistantly on the circumference of a not depicted circle on a desktop display and were rotated 60 degrees clockwise during the exposure phase. One group pointed at targets crossing their body midline, the other group pointed at targets not crossing their body midline. The results suggest that the retention of adaptive behaviour is weakened when the pointing movements cross the body midline. Barral and Debû (2004) did not focus on adaptation but on pointing accuracy, sex, and the preferred and non-preferred hand of right-handed participants. They found a leftward bias for the left hand and the opposite effect for the right hand (Barral and Debû 2004). In our case, this would only explain the rightward bias found for targets on the right side, e.g. during the baseline phase, but contradicts the leftward bias found for targets positioned on the left side. These two studies could partly explain that pointing accuracy could differ for targets positioned on the left or right due to midline crossing or handiness. However, both theories do not explain a horizontal outward deviation in both directions for all phases without concurrent movement feedback.
6 Conclusion and outlook
The present experiment investigated the influence of instruction and body representation on magnitude and persistence of the aftereffect following sensorimotor adaptation in VR. The instruction was manipulated via false information about a random error component and influenced the persistence. This demonstrates that cognitive factors should be considered, for example, in VR training. Further research could investigate, for instance, whether the instruction can influence effects of actual variability of visual error on the aftereffect. In this experiment, only negative expectations of the VR system were investigated. There were two reasons for this. First, with VR, it is generally more difficult to convince the user of a perfect application than of a faulty application. Second, the sample consisted of students who are generally very interested in technology, some of whom already have experience with VR and know that a flawless application is an exaggeration. Future studies could investigate the impact of positive expectations, e.g. grandiose capabilities of the device, on adaptation and aftereffects in VR.
No effects of body representation were found, but the results indicate that individual perception of realism may play a role. Since VR is becoming more and more realistic and at the same time can offer a novel experience in relation to one's own body, future studies could consider realism and unfamiliarity as possible factors influencing sensorimotor adaptation. Finally, a systematic movement error, the target position effect, was exploratively found, which also offers starting points for further research, e.g. whether this effect depends on the type of pointing movement (hand vs. cursor).
Availability of data and material
The data for the experiment is available under https://doi.org/10.14279/depositonce-17735.
Code availability
The unity code for the experiment can be provided by the authors upon request.
Notes
The effects size \({d}_{\psi }\) should always be interpreted according to the linear combination of the model parameters. Please note that Cohen’s d conventions do not apply here (Kline 2013).
References
Anglin JM, Sugiyama T, Liew S-L (2017) Visuomotor adaptation in head-mounted virtual reality versus conventional training. Sci Rep 7:45469. https://doi.org/10.1038/srep45469
Aziz JR, MacLean SJ, Krigolson OE, Eskes GA (2020) Visual feedback modulates aftereffects and electrophysiological markers of prism adaptation. Front Hum Neurosci 14:138. https://doi.org/10.3389/fnhum.2020.00138
Banakou D, Groten R, Slater M (2013) Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes. Proc Natl Acad Sci USA 110:12846–12851. https://doi.org/10.1073/pnas.1306779110
Baron RM, Kenny DA (1986) The moderator–mediator variable distinction in social psychological research: conceptual, strategic, and statistical considerations. J Pers Soc Psychol 51:1173–1182. https://doi.org/10.1037/0022-3514.51.6.1173
Barral J, Debû B (2004) Aiming in adults: sex and laterality effects. Laterality 9:299–312. https://doi.org/10.1080/13576500342000158
Bedford FL (1989) Constraints on learning new mappings between perceptual dimensions. J Exp Psychol Hum Percept Perform 15:232–248. https://doi.org/10.1037/0096-1523.15.2.232
Bedford FL (2001) Towards a general law of numerical/object identity. Curr Psychol Cogn 20:113–175
Biocca FA, Rolland JP (1998) Virtual eyes can rearrange your body: adaptation to visual displacement in see-through, head-mounted displays. Presence 7:262–277. https://doi.org/10.1162/105474698565703
Burin D, Kilteni K, Rabuffetti M et al (2019) Body ownership increases the interference between observed and executed movements. PLoS ONE 14:e0209899. https://doi.org/10.1371/journal.pone.0209899
Cho S, Chang WK, Park J et al (2022) Feasibility study of immersive virtual prism adaptation therapy with depth-sensing camera using functional near-infrared spectroscopy in healthy adults. Sci Rep 12:767. https://doi.org/10.1038/s41598-022-04771-5
Clower DM, Boussaoud D (2000) Selective use of perceptual recalibration versus visuomotor skill acquisition. J Neurophysiol 84:2703–2708. https://doi.org/10.1152/jn.2000.84.5.2703
Cohn BA, Maselli A, Ofek E, Gonzalez-Franco M (2020) SnapMove: movement projection mapping in virtual reality. In: 2020 IEEE international conference on artificial intelligence and virtual reality (AIVR). IEEE, Utrecht, Netherlands, pp 74–81
Valve Corporation (2023) Valve Index Headset, Bellevue
Della-Maggiore V, Landi SM, Villalta JI (2015) Sensorimotor adaptation: multiple forms of plasticity in motor circuits. Neuroscientist 21:109–125. https://doi.org/10.1177/1073858414545228
Dewar R (1970) Adaptation to displaced vision: amount of optical displacement and practice. Percept Psychophys 8:313–316. https://doi.org/10.3758/BF03212599
Doerner R, Broll W, Jung B et al (2022) Introduction to virtual and augmented reality. In: Doerner R, Broll W, Grimm P, Jung B (eds) Virtual and augmented reality (VR/AR). Springer International Publishing, Cham, pp 1–37
Fernandes HL, Stevenson IH, Kording KP (2012) Generalization of stochastic visuomotor rotations. PLoS ONE 7:e43016. https://doi.org/10.1371/journal.pone.0043016
Fernández-Ruiz J, Díaz R (1999) Prism adaptation and aftereffect: specifying the properties of a procedural memory system. Learn Mem 6:47–53. https://doi.org/10.1101/lm.6.1.47
Fleury L, Prablanc C, Priot A-E (2019) Do prism and other adaptation paradigms really measure the same processes? Cortex 119:480–496. https://doi.org/10.1016/j.cortex.2019.07.012
Fox J (2016) Applied regression analysis and generalized linear models, 3rd edn. SAGE, Los Angeles
Gammeri R, Turri F, Ricci R, Ptak R (2020) Adaptation to virtual prisms and its relevance for neglect rehabilitation: a single-blind dose-response study with healthy participants. Neuropsychol Rehabil 30:753–766. https://doi.org/10.1080/09602011.2018.1502672
Gonzalez EJ, Chase EDZ, Kotipalli P, Follmer S (2022) A model predictive control approach for reach redirection in virtual reality. In: CHI conference on human factors in computing systems. ACM, New Orleans LA USA, pp 1–15
Gonzalez-Franco M, Cohn B, Ofek E et al (2020) The self-avatar follower effect in virtual reality. In: 2020 IEEE conference on virtual reality and 3d user interfaces (VR). IEEE, Atlanta, GA, USA, pp 18–25
Haans A, IJsselsteijn WA, De Kort YAW (2008) The effect of similarities in skin texture and hand shape on perceived ownership of a fake limb. Body Image 5:389–394. https://doi.org/10.1016/j.bodyim.2008.04.003
Havermann K, Lappe M (2010) The Influence of the consistency of postsaccadic visual errors on saccadic adaptation. J Neurophysiol 103:3302–3310. https://doi.org/10.1152/jn.00970.2009
Juliano JM, Schweighofer N, Liew S-L (2022) Increased cognitive load in immersive virtual reality during visuomotor adaptation is associated with decreased long-term retention and context transfer. J NeuroEngineering Rehabil 19:106. https://doi.org/10.1186/s12984-022-01084-6
Kilteni K, Groten R, Slater M (2012a) The sense of embodiment in virtual reality. Presence 21:373–387. https://doi.org/10.1162/PRES_a_00124
Kilteni K, Normand J-M, Sanchez-Vives MV, Slater M (2012b) Extending body space in immersive virtual reality: a very long arm illusion. PLoS ONE 7:e40867. https://doi.org/10.1371/journal.pone.0040867
Kline RB (2013) Beyond significance testing: statistics reform in the behavioral sciences. American Psychological Association, Washington, DC
Kornheiser AS (1976) Adaptation to laterally displaced vision: a review. Psychol Bull 83:783–816. https://doi.org/10.1037/0033-2909.83.5.783
Michel C, Pisella L, Prablanc C et al (2007) Enhancing visuomotor adaptation by reducing error signals: single-step (aware) versus Multiple-step (unaware) exposure to wedge prisms. J Cogn Neurosci 19:341–350. https://doi.org/10.1162/jocn.2007.19.2.341
Nilsson NC, Peck T, Bruder G et al (2018) 15 years of research on redirected walking in immersive virtual environments. IEEE Comput Grap Appl 38:44–56. https://doi.org/10.1109/MCG.2018.111125628
Norris SA, Greger BE, Martin TA, Thach WT (2001) Prism adaptation of reaching is dependent on the type of visual feedback of hand and target position. Brain Res 905:207–219. https://doi.org/10.1016/S0006-8993(01)02552-5
O’Shea J, Gaveau V, Kandel M et al (2014) Kinematic markers dissociate error correction from sensorimotor realignment during prism adaptation. Neuropsychologia 55:15–24. https://doi.org/10.1016/j.neuropsychologia.2013.09.021
Prablanc C, Panico F, Fleury L et al (2020) Adapting terminology: clarifying prism adaptation vocabulary, concepts, and methods. Neurosci Res 153:8–21. https://doi.org/10.1016/j.neures.2019.03.003
R Core Team (2023) R: a language and environment for statistical computing. R Core Team, Vienna
Radeau M, Bertelson P (1977) Adaptation to auditory-visual discordance and ventriloquism in semirealistic situations. Percept Psychophys 22:137–146. https://doi.org/10.3758/BF03198746
Redding GM, Wallace B (2006) Generalization of prism adaptation. J Exp Psychol Hum Percept Perform 32:1006–1022. https://doi.org/10.1037/0096-1523.32.4.1006
Renner RS, Velichkovsky BM, Helmert JR (2013) The perception of egocentric distances in virtual environments: a review. ACM Comput Surv 46:1–40. https://doi.org/10.1145/2543581.2543590
Rolland JP, Biocca FA, Barlow T, Kancherla A (1995) Quantification of adaptation to virtual-eye location in see-thru head-mounted displays. In: Proceedings Virtual Reality Annual International Symposium ’95. IEEE Comput. Soc. Press, Research Triangle Park, NC, USA, pp 56–66
Slater M, Perez-Marcos D, Ehrsson HH, Sanchez-Vives MV (2009) Inducing illusory ownership of a virtual body. Front Neurosci 3:214–220. https://doi.org/10.3389/neuro.01.029.2009
Thomas M (2012) The effect of different movement exercises on cognitive and motor abilities. APE 02:172–178. https://doi.org/10.4236/ape.2012.24030
Tsakiris M (2010) My body in the brain: a neurocognitive model of body-ownership. Neuropsychologia 48:703–712. https://doi.org/10.1016/j.neuropsychologia.2009.09.034
Veilleux L-N, Proteau L (2015) Prism adaptation in virtual and natural contexts: evidence for a flexible adaptive process. Q J Exp Psychol 68:1168–1182. https://doi.org/10.1080/17470218.2014.977305
Vetter P, Goodbody SJ, Wolpert DM (1999) Evidence for an eye-centered spherical representation of the visuomotor map. J Neurophysiol 81:935–939. https://doi.org/10.1152/jn.1999.81.2.935
Wähnert S, Gerhards A (2022) Sensorimotor adaptation in VR: magnitude and persistence of the aftereffect increase with the number of interactions. Virtual Real 26:1217–1225. https://doi.org/10.1007/s10055-022-00628-4
Wei K, Körding K (2009) Relevance of error: What drives motor adaptation? J Neurophysiol 101:655–664. https://doi.org/10.1152/jn.90545.2008
Welch RB (1971) Prism adaptation: The “target-pointing effect” as a function of exposure trials. Percept Psychophys 9:102–104. https://doi.org/10.3758/BF03213039
Welch RB (1972) The effect of experienced limb identity upon adaptation to simulated displacement of the visual field. Percept Psychophys 12:453–456. https://doi.org/10.3758/BF03210933
Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88:638–667. https://doi.org/10.1037/0033-2909.88.3.638
Wilf M, Cerra Cheraka M, Jeanneret M et al (2021) Combined virtual reality and haptic robotics induce space and movement invariant sensorimotor adaptation. Neuropsychologia 150:107692. https://doi.org/10.1016/j.neuropsychologia.2020.107692
Wilf M, Dupuis C, Nardo D et al (2023) Virtual reality-based sensorimotor adaptation shapes subsequent spontaneous and naturalistic stimulus-driven brain activity. Cereb Cortex 33:5163–5180. https://doi.org/10.1093/cercor/bhac407
Wilke C, Synofzik M, Lindner A (2013) Sensorimotor recalibration depends on attribution of sensory prediction errors to internal causes. PLoS ONE 8:e54925. https://doi.org/10.1371/journal.pone.0054925
Acknowledgements
We are grateful to our colleague Richard Gross, PhD, for programming the experiments.
Funding
Open Access funding enabled and organized by Projekt DEAL. The article processing charge (APC) were funded by the joint publication funds of the TU Dresden, including Carl Gustav Carus Faculty of Medicine, and the SLUB Dresden as well as the open access publication funding of the DFG. No other funds, grants, or support was received.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by US. The first draft of the manuscript was written by SW, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Ethics approval
The study was approved by the research ethics committee of the Department of Psychology and Ergonomics at Technische Universität Berlin.
Consent to participate
All participants gave their informed consent prior to data collection.
Consent for publication
All authors gave their consent for publication.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Wähnert, S., Schäfer, U. Sensorimotor adaptation in virtual reality: Do instructions and body representation influence aftereffects?. Virtual Reality 28, 47 (2024). https://doi.org/10.1007/s10055-024-00957-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10055-024-00957-6