Introduction

Manipulating objects require the generation of a sequence of motor commands together with the prediction of their sensory consequences. Tasks involving object manipulation rely on multiple sensory systems (e.g., auditory, visual, tactile, and kinesthetic) to generate an appropriate hand trajectory. This trajectory has to take into account the initial state of the motor apparatus, object properties, and object and target location if goal-directed behavior is to succeed (Flanagan et al. 2006; Jeannerod 1986). For a single grasp movement, one can identify separate task units such as the determination of the target location; the transformation of the visual information on target location into a reference frame suitable for planning the action; and the initiation and control of the movement including grip formation, prehension, and a subsequent transport movement to a new target (Desmurget et al. 1998; Land 2009; Smeets and Brenner 2006). The relationship between gaze behavior and kinematics plays an essential role in such transport movements (Flanagan et al. 2006; Jeannerod 1986; Jeannerod et al. 1995).

Previous research has shown that gaze behavior is determined primarily by the spatial location of the information needed for the ongoing or next action (Land 2009; Land and Tatler 2009). Accordingly, eye movements can be used (1) to maintain a stable fixation on the grasped object to monitor and, if necessary correct the ongoing transport movement; and (2) to fixate the target location of the transport phase before the hand reaches the target to gain spatial information that is then transformed into motor commands to increase accuracy (Flanagan et al. 2006; Jeannerod 1986; Johansson et al. 2001). Hence, if the grasping of an object is followed by its transport to a new target location, two competing gaze locations will emerge: that required to control the transport movement and that required for an anticipative saccade to the new target location. When gaze is fixated on the object during transport, the projection of the object on the retina remains stable due to smooth eye movements. At the same time, the target’s projection on the retina moves from the periphery to the center. After the saccade to the target, the target’s projection on the retina remains stable, and the object’s projection approaches from the periphery to the center. In both cases, the distance between object and target is represented in a direct way.

By studying sequences of grasping and transport movements instead of a single grasp-to-transport movement, it also becomes possible to analyze gaze behavior during the release between different actions. This movement phase is characterized by the need to monitor the outcome of the previous action (‘evaluation fixation’), and the preparation/planning of the next action at the same time. When one action is finished, its result will be controlled visually while the next action is being initiated resulting in a rivalry between the visual control of the completed action and the anticipatory requirements for the next action. In sequential tasks, gaze is occasionally directed to the next target before the preceding target is reached by the hand (Ballard et al. 1992; Land et al. 1999). If the completion of the first task is challenging, it can be expected that gaze will stay longer on the object(s) of the current task than on fixating the new target following the subsequent predictive saccade to the new target (Bowman et al. 2009).

In the present study, we aim to investigate how task difficulty modulates gaze behavior with respect to the balance between visually monitoring the ongoing action and prospectively collecting visual information about the future course of the ongoing action. To the best of our knowledge, only a limited number of studies adopting a task difficulty approach have examined gaze behavior together with movement kinematics during complex everyday movements (see Latash and Jaric 2002; Rand and Stelmach 2011), and no study has varied task difficulty within movement sequences. Therefore, we examined sequences of reach and transport movements of water glasses that differed in task difficulty in terms of different filling levels. One might imagine these sequences as part of a waiter’s action routine. This procedure allowed us to analyze not only typical parameters of grasping behavior (Desmurget et al. 1998; Jeannerod 1986) but also the accompanying gaze behavior in a more naturalistic setting (Hayhoe and Ballard 2005; Smeets et al. 1996). Furthermore, it enabled us to more closely examine the balance between the requirement to monitor an ongoing action and to collect information about future course of the action from fixating task-relevant locations in the environment.

To gain insights into the way task difficulty modulates the sensorimotor system in balancing the need of reactive action monitoring and prospective action control, we used a sequential grasping-to-transport task and focused our analyses on two movement phases:

  1. (a)

    the transport phase is the time interval between the point in time at which the object had been grasped until it was placed on the respective target location. Here, the sensorimotor system has to weigh the time spent on visually tracking the object during transport against the time spent on visually fixating the target location where the object had to be placed.

  2. (b)

    the release phase is the time interval between the point in time at which the object had been released after it had been placed on the target and the reaching of the next object in the sequence. Here, the sensorimotor system has to weigh the time spent on visually monitoring the action outcome against the time spent on visually fixating the next object that had to be reached.

For the transport phase, we expected longer absolute durations of online gaze control with increasing task difficulty which can result in one of two possible pattern of gaze behavior. First, we expected that the absolute time participants focus their gaze on the glass will increase and the absolute time participants focus their gaze on the target will not change. This will lead to an increase of the relative time of the gaze on the glass and a decrease of the relative time of the gaze on the target. Second, we expected that the absolute time participants focus their gaze on the glass and on the target will increase proportionally. This will lead to no change of the relative time of the gaze on the glass and on the target. Additionally, we aimed to characterize the interplay between gaze behavior and movement kinematics to uncover relevant relations due to task difficulty requirements.

For the release phase, we hypothesized a difficulty-dependent persistency of the evaluation fixation while the hand starts to move toward the next object. In more specific terms, we expected that the duration of the release phase as well as the fixation duration on the glass after release increased with higher task difficulty.

Methods

Participants

Ten self-reported right-handed participants, two men, and eight women, took part in this study. Their age ranged from 18 to 26 years (M = 20.8, SD = 2.4). Eight participants had normal vision and two participants had corrected to normal vision using contact lenses. The study complied with the Declaration of Helsinki. Before the experiment started, all participants gave written informed consent and they were naïve to the aim of the study.

Design and procedure

Participants were seated centrally at a table in a well-illuminated room. They had to perform ten series of transport movements of water glasses with different filling levels of 100, 94, 88, 82, and 76% (Fig. 1) on two horizontal planes (see Fig. 2). Within each series, participants had to grasp the glass on the table (A) and transport it to its goal on the same plane (gray circle). Directly after that, they had to grasp the glass (B) on the upper plane and transport it to the target on the same plane (gray circle). Then participants grasped the glass on the table (A) and transported it to the starting spot. They repeated the same procedure for the glass on the upper plane (B). This was repeated 4 times resulting in 16 trials per session, 8 on each plane. Each participant made a total of 160 transport movements. Grasp movements followed generally to the form of a precision grip, though, a precision grip in our study involved not only to use the index finger and the thumb. In line with the definition by Napier (1956, p. 904) “the object may be pinched between the flexor aspects of the fingers and the opposing thumb”.

Fig. 1
figure 1

Schematic illustration of the water glasses with different filling levels (100, 94, 88, 82, and 76%). The filled circles represent thumb–glass contact and the filled squares represent index finger–glass contact

Full size image
Fig. 2
figure 2

Schematic illustration of experimental setup

Full size image

In all trials, we used a 130 × 65 mm glass with a weight of 300 g; and in each single session, we used two different levels of difficulty. Because the specific tasks varied only slightly in terms of weight, we expected only small differences in movement trajectories due to differences in mass (Eastough and Edwards 2007). Combinations of difficulty were counterbalanced between participants. We instructed them to transport the glasses continuously from one spot to the other. There was no special instruction not to spill water and they could perform the trials at a self-paced speed. In under 1% of all trials, using a completely filled glass, (100%) participants spilled water, whereas during all other trials, using different filling levels (94, 88, 82, 76%), participants did not spill water.

Data recording

Gaze behavior was recorded at 30 Hz with a mobile head-mounted eye tracker (SMI iViewETG, SensoMotoric Instruments, Germany) using iViewETG (Ver. 2.1) software. The eye tracker consists of three cameras which are built into the goggles. One high definition video camera recorded the environment (960 × 720 pixels) and two infrared cameras recorded the eye movements of each eye separately. Data were stored on a laptop (Lenovo X230, ThinkPad, USA) connected to the eye tracker via USB. Before starting the experiment, we conducted a three-point calibration. We determined gaze position to an accuracy of about 0.5° and the spatial resolution of the system to an accuracy of about 0.1°.

Kinematic data were recorded by means of a motion capture system (VICON, Oxford, England) with a frame rate of 50 Hz. Three markers were attached on top of the glass. Additional markers were attached to the thumb, index finger, and wrist of the right hand.

Before starting each session, an external trigger signal was recorded by both systems, the eye tracker and the motion capture system, to synchronize gaze data and kinematics.

Data analysis

First, we downsampled the kinematic data to the frame rate of the eye tracker (30 Hz) using linear interpolation to allow joint post-processing of gaze and kinematic data. We preprocessed the kinematic data with Nexus 1.8.5 (VICON, Oxford, England). All further analyses were carried out with MATLAB R2015a (MathWorks, Natick, MA, USA). Data were filtered with a fourth-order Butterworth low-pass filter with a cut-off frequency of 6 Hz. Afterward, we determined the velocity curve of the glass marker. Velocities were applied to segment each single trial into four task units, however, we concentrated the analyses on later phases, transport and release. (1) Reach phase period from starting to reach for the glass until starting to grasp it. The start was defined as the point when the velocity of the wrist marker exceeded a threshold of 30 mm/s (Brouwer et al. 2009). The end was defined as the point when the velocity of the wrist marker dropped below this threshold. (2) Grasp phase period starting with the end of the reach phase and ending when the velocity of the glass marker exceeded the threshold as defined before. (3) Transport phase period between starting to transport the glass and placing it on the target. The same velocity threshold as used for grasp was utilized to define the start and end of the transport phase. (4) Release phase period between the end of the transport phase and the beginning of the next reach phase.

Concerning the task units and for the sake of completeness, we determined the following kinematic parameters (see Appendix 1 in ESM): (1) reach movement duration, peak aperture; (2) grasp movement duration, distance between the rim of the glass and the thumb and between the rim of the glass and the index finger; (3) transport movement duration, relative time of the peak velocity, peak trajectory height, and length of trajectory; (4) release movement duration.

Two independent observers analyzed gaze data frame by frame. The following areas of interest (AOIs) were defined to localize the gaze in the visual scene: (1) glass, (2) target location at which the glass had to be placed, and (3) all other task-irrelevant locations. A total of 1.7% of the data was missing (e.g., blinking). The following dependent variables were examined: (1) cumulative percentual duration of time focusing on each AOI during transport; (2) time interval between the first gaze on the glass and the start of transport; (3) relative and absolute time when the saccade from the glass to the target occurred; (4) appearance and duration of an evaluation fixation (an evaluation fixation was defined as the gaze directed toward the top of the glass after the transport phase; e.g., to check whether water was spilled).

No significant effects were identified for the direction of the movements (left to right vs. right to left) for all filling levels. Therefore, we collapsed the data on movement direction. Statistical analyses were based on repeated measures ANOVAs. Corrections were made with Greenhouse–Geisser. In all tests, the level of significance was set at p < 0.05. Post hoc analyses were calculated using t tests with Bonferroni correction.

Results

The complete sequential motor task can be represented as a series of 16 action sequences, each consisting of 4 task units. We will report, primarily, gaze data as addressed in the research question and for the respective hypotheses, but for the sake of completeness, we will outline also kinematic data when it is necessary for a valid interpretation of the whole dataset (a more complete description concerning kinematic data can be found in ESM Appendix 1 and for gaze data in ESM Appendix 2).

As a prerequisite for the examination of gaze parameters relying on task difficulty, we analyzed the kinematic data using univariate ANOVAs with repeated measures for the factor filling level (see Appendix 1 in ESM). Higher filling levels (100 and 94%) had a significant impact on most parameters resulting in, for example, longer movement durations and smaller distances between the thumb/index finger and the rim. However, results did not differ significantly between the lower filling levels (88, 82, and 76%).

Because our main interest was in the interaction between the hand movements and the accompanying gaze, we partitioned the action sequences into four task units (reach, grasp, transport, and release) that fulfill different functions in the completion of the task. Thereby, we will focus on the transport and the release interval to investigate how task difficulty affects the sensorimotor system.

Grasp phase

As reported in ESM Appendix 1, task difficulty affected the duration of the grasp phase with longer durations for higher filling levels. Filling level also influenced grip position. A 2 (finger: thumb vs. index finger) × 5 (filling level: 100 vs. 94 vs. 88 vs. 82 vs. 76%) ANOVA with repeated measures for both factors was applied to analyze the distance of fingers to the rim. Results (see Fig. 1) showed a significant main effect for filling level, F (4,36) = 15.58, p < 0.001, indicating a closer hand–glass contact to the rim for higher filling levels. This result demonstrates a predictive effect for the reach–to–grasp phase. A modification of the grip position has functional relevance for the following transport of the glass because the transport is expected to be more smooth, if the grasp is closer to the rim. Therefore, taking into account task difficulty features already in the grasping phase does reflect motor requirements for the following transport phase.

Transport phase

The percentage viewing time on the AOIs during the transport phase is illustrated in Fig. 3. A 3 (area: glass vs. target vs. other) × 5 (filling level: 100 vs. 94 vs. 88 vs. 82 vs. 76%) ANOVA with repeated measures for both factors revealed a significant main effect for Area, F(2,18) = 84.69, p < 0.001, due to a longer viewing time on the glass (M = 42.3%) and on the target (M = 45.2%) compared to other (M = 5.9%). Results also showed a significant Area × Filling level interaction, F(8,72) = 44.51, p < 0.001, due to a longer viewing time on the glass and a shorter viewing time on the target for higher filling levels.

Fig. 3
figure 3

Percentage viewing time on the areas of interest (glass, target, and other) during transport for different filling levels (100, 94, 88, 82, 76%). Error bars indicate standard error

Full size image

We also found that the absolute time interval between the first gaze toward the glass and the start of the transport phase, F(4,36) = 42.35, p < 0.001, was significantly longer for higher filling levels (see Fig. 4). This resulted primarily from absolute longer reach and absolute longer grasp phases (see Appendix 1 in ESM), whereas the point in time (see Fig. 4) when the first gaze on the glass occurred was hardly affected.

Fig. 4
figure 4

Sequences of eye and hand movements. Mean duration of task units (Re reach, G grasp, T transport, R release) and mean duration of gaze on areas of interest are shown. The last gaze on the glass (dotted lines) occurred on 59.8% of all trials for 76% filling level, 63.3% of all trials for 82% filling level, 59.0% of all trials for 88% filling level, 84.0% of all trials for 94% filling level, and 90.6% of all trials for 100% filling level. The second reach phase indicates that subjects already started the next reach movement because they had to perform a sequence of movements

Full size image

A separate ANOVA revealed that the absolute time between the start of the saccade from the glass to the target and the end of the transport phase did not differ significantly, F(4,36) = 1.11, p = 0.37 (see Fig. 5b). In contrast, Fig. 5a illustrates that the relative time between the start of the saccade from the glass to the target and the end of the transport phase was significantly smaller for higher filling levels, F(4,36) = 27.08, p < 0.001. This result shows that the gaze shift is independent from task difficulty and is triggered in a way, that lead to a more or less constant point in time relating to the end of the transport.

Fig. 5
figure 5

a Relative time and b absolute time from the start of the saccade from the water glass to the target until the end of the transport phase for different filling levels (100, 94, 88, 82, and 76%). Means and standard errors are reported

Full size image

Release phase

The kinematics during release and the subsequent new reach together with the related gaze behavior revealed a gaze pattern for evaluating the “outcome” of the release component on a large number of trials. Subjects often made an evaluation fixation on the top of the glass to check whether or not water had been spilled. An ANOVA for filling level showed that the start of the evaluation fixation did not differ between filling levels, F(4,36) < 1, ns. The evaluation fixation started, irrespective of filling level, about 80 ms after the start of the release phase.

There was a significantly higher proportion of evaluation fixations for higher filling levels, χ 2(4, N = 1600) = 153.98, p < 0.001, as well as a significantly longer absolute duration of the evaluation fixation for higher filling levels, F(4,36) = 32.27, p < 0.001.

To analyze the influence of the evaluation fixation on the subsequent reach phase, we calculated a correlation coefficient between the duration of the evaluation fixation and the reach duration. Correlation coefficients were first calculated for each subject separately and then averaged after a Fisher z transformation. This parameter (r = 0.05) did not reveal a significant effect.

Discussion

Task difficulty, as operationalized by the filling level of a water glass, influenced both gaze behavior and kinematics in sequences of reach–to–grasp and transport movements. This study utilized a more general aspect of task difficulty that goes beyond the traditional view as defined by Fitts’s law (Latash and Jaric 2002). The present results reveal that task difficulty modulates the balance between reactive action monitoring and prospective action control defined by the transport and release phases of the movement sequences, respectively. Because we examined sequences of grasping and transport movements, we had to consider not only a typical reach–grasp–transport–release sequence but also a release-to-reach sequence. Moreover, a reach does not start from a defined resting position (except for the first reach) but “out of the movement” and immediately after the release of the previous water glass. Therefore, the task contains a continuously dynamic interaction within and between functionally different action units of subsequent action sequences and particularly requires a coordinated interplay of gaze behavior and movement kinematics.

The present results reveal a significant influence of task difficulty on the interaction between gaze and kinematics during transport and a strong influence of task difficulty on gaze during the release phase between different grasp-to-place movements. Higher filling levels are characterized by significantly longer absolute movement durations, and significantly longer percentage viewing times of the transported object during the transport phase, and significantly more and absolute longer evaluation fixations during the release phase. This suggests that ecologically valid movements with higher task difficulty elicit a systematic change in the control mechanism of goal-directed manual actions (closed loop) (cf. Appendix 1 in ESM). Considering a single sequence of the grasp and transport movements, task difficulty unfolded most of its impact during the later stages, that is during transport and release. Greater impact is due to comparatively higher precision requirements and longer movement durations.

Grasp

Regarding grasping phase, task difficulty impacted mainly on the kinematics and less on gaze behavior. That is because regardless of filling level gaze focused consistently on the object during that movement phase. Due to the intrinsic logic of the task requirements and the short absolute duration of phase, it seems reasonable for task difficulty to lead to no differences in gaze as well as in kinematics. However, as ESM Appendix 1 shows, the absolute duration of grasp for the 100% and the 94% level differed by about 140%. This shows clearly that task difficulty is represented proactively during the approach of the hand. An additional kinematic parameter sensitive to task difficulty is the proximity of fingers to the rim, even though these changes seem to be quite small. Changes in the finger configuration are consistent with findings on grasping points when object features are changed (Eastough and Edwards 2007; Voudouris et al. 2010). In the present case, the shape of the object was held constant, and mass varied only to a very small degree. However, placing the fingertips slightly closer to the rim makes it possible to control the water glass more in a pendulum-like manner. This feature does not seem absolutely essential for the control of grip itself, but it is useful for the subsequent transport phase as a “built-in” strategy to compensate for fluctuations and thereby guarantees for a high degree of consistency in the movement outcome and thus avoid spilling water. It, therefore, reflects a predictive component for the subsequent transport phase, too.

Transport

For transport, absolute duration of the sequence did differ with task difficulty which seems to be trivial. Again, major differences were found for the two highest filling levels. For gaze behavior, the main result is that the higher the filling level, the longer the percentage time on glass and the shorter the percentage time on target. Task difficulty has a major impact on the percentage viewing time for relevant cues, and therefore affects the predictive saccade to the target (Flanagan et al. 2006; Jeannerod 1986; Johansson et al. 2001). It has to be considered that the timing of both the arm movements and the eye movements is closely linked. It is, therefore, plausible to assume that the motor program for the arm movement also includes features for the occular motor system during the same task (Flanagan and Johansson 2003; Johansson et al. 2001) and vice versa. Due to task difficulty-dependent longer absolute movement durations for transport, we observed two interesting and interrelated findings concerning this saccade. First, the relative time until the saccade appeared increased with task difficulty, that is for higher filling levels, the saccade appears relatively later. This effect is accompanied by longer absolute movement durations, so that the water level (prevention of spillage) is controlled for a disproportionately longer time. On the other hand, the absolute durations between the saccade and the placing of the object on the target seems to be rather similar (Fig. 5b). Independently from higher task difficulty and longer absolute movement durations, the saccade starts between 700 and 800 ms before object release. This can be interpreted as evidence for the existence of a timing mechanism that continuously calculates the time left to complete the action. This finding fits well with assumptions of time-to-contact research postulating motor control mechanisms that adjust the position-time curves of targeted movements just to the end point (Savelsbergh et al. 1991). At present, however, we cannot distinguish whether this mechanism is predictive or reactive in nature. In a reactive timing mechanism, time to movement termination would not have to be explicitly computed. Rather, the occurrence of the target on a particular area centered around the fovea could serve as a trigger for a saccade to the target. Alternatively, an internal forward model could use efference copy signals and sensory information about the current state of the body to explicitly compute the time left to complete the movement. Once this time drops below a certain threshold, a saccade to the placement target would be triggered to facilitate homing in on the target. At present, we are unable to distinguish these two possibilities.

Release

To the best of our knowledge, the occurrence of an evaluation fixation during release and the subsequent reach to another object has not been described in detail before. We observed an increase in the proportion of such evaluation fixations and their absolute durations for higher task difficulties. Participants fixated the rim of the glass further on even when opening of the fingers was finished and the movement of the hand to a different object has started. However, the task difficulty of the next object did not influence neither (the absolute duration of) the evaluation fixation nor the kinematics of the next reach. Taken as a whole, these observations show that the successful release of the water glass (without water spilling) and the validation of that homing phase take first priority when task difficulty increases. This task-dependent evaluation process even seems to contain processes that circumvent typical predictive movement and gaze behavior as seen in nonsequential tasks (Flanagan and Johansson 2003; Rotman et al. 2006). Although we found a gaze pattern as reported in other reach and grasp studies (Hayhoe and Ballard 2005; Land and McLeod 2000; Land et al. 1999), our findings in particular stresses the need of a coordinated interplay of motor and gaze control in reach-to-grasp movements (Latash and Jaric 2002).

Previous work has concentrated primarily on the issue of action planning. This has been documentated mainly by predictive saccades to the target(s) during grasping (Flanagan et al. 2006; Johansson et al. 2001). We can now confirm this finding for reaching movements, too. Furthermore, for the release phase and the transition from release to the subsequent reach phase, we discovered a previously unreported gaze behavior that is characterized by ‘evaluation’ fixations, whose function obviously is to control for the successful completion of the release-to-reach transition. Unexpectedly, this kind of gaze behavior was affected only by the task difficulty of the previous movement sequence but not of that of the subsequent one. This pattern demonstrates that prediction is a sine qua non-requirement for object manipulation, but that the task-dependent (feedback-based) control of the ongoing movement and the evaluation of the intended movement effect are also highly significant.

In general, we found a movement and gaze pattern revealing an influence of task difficulty that was especially evident for the later phases of transport and release. A coordinated interplay of movement formation and gaze behavior based on visual and motor anchor points (Bowman et al. 2009; Neggers and Bekkering 2000) in these sections seems to be important for the successful mastery of sequential reach-to-grasp tasks (see Henriques et al. 1998).