Perceptual Illusion and Development of a Sense-Centered Human Interface

Abstract

Tele-existence is the replication of physically plausible information through the provision of real sensation of presence. Here, we sought to elucidate the mechanisms of perceptual illusion within the context of brain function. Improving our understanding of perceptual illusion will contribute to the realization of new and more efficient human interfaces.

7.1 Introduction

Our bodies contain numerous and various sensors. Those sensors send signals to the cerebral cortex, and sensations are then perceived. Most sensations are identified with a specific type of stimulus. Furthermore, perception is not directly related to sensor activity. Most sensors receive signals passively. Muscles act as actuators to create the force for body motion, but the sensors also detect force and positioning of the body. Our hands have many tactile sensors and can be controlled voluntarily. So ours hands are active sensor for touch.

Computer input devices, such as mice and trackpads, measure hand position and translate it to cursor position on a screen. Recently, Apple$^{{\textregistered }}$ developed the Force Touch trackpad, adding a new dimension to touch interfaces. New multi-touch gestures with force are being developed and adopted in computer interaction. However, we control force in touching unconsciously, so gestures have to be developed with consideration to behavior in daily life. Muscle activity reflects changes in force and can be measured as electromyography (EMG) signals . Proposed here is a new interface which measures muscle activity and estimates not only position, but also force and stiffness simultaneously. To develop intuitive manipulation, muscle control for manipulation was also studied.

When we manipulate an object, its weight is important information in the production of joint torque, because the dynamics of the object are non-linear and complex. Also, the perception of heaviness is still an open problem, and the size-weight illusion, where small objects feel heavier than large objects of the same weight, is a well known phenomenon.

In this chapter, a sensor which detects weight is discussed, and a new illusion in perception is explained. The musculoskeletal model plays a crucial role in understanding the phenomenon. A new human interface based on the musculoskeletal model is also introduced.

Table 7.1 Sensory systems

7.2 Haptics and Force

Sensory systems measure various stimuli and perceive information from the environment. Table 7.1 shows the different types of sensory systems and their associated modalities of information [1]. Our sensory systems use four types of sensory receptors, mechanoreceptors, chemoreceptors, photoreceptors, and thermoreceptors. Chemoreceptors are related to olfaction, gustation, itch, and visceral sensations. Visual sensors detect light, with different visual sensors reacting to different wavelengths. The brain integrates the signals from receptors and extracts information, such as smell, color, or temperature.

As shown in Table 7.1, there is no sensor to directly measure weight. Somatosensory information is received at the hand by pressure, as well as at the arm by hand displacement, and captured by cutaneous mechanoreceptors (touch) or muscle and joint receptors (proprioception). Muscles have three kinds of mechanoreceptors, which respond to muscle length, contraction velocity, and muscle force.

Proprioception refers to the sensing of information from muscle and joint receptors on the body’s own motion. The sensing of one’s own motion is called kinaesthesia, and the main receptor involved in kinaesthesia is the muscle spindle. Motor commands from the central nervous system (CNS) related to the perception of heaviness are sent to the muscles. The receptors also send afferent feedback signals back to the CNS. The CNS can estimate the result of the motor command using an efference copy and compare the estimate with the afferent feedback.

Exteroception refers to information from outside of the body, such as light, sound, and smell. Vision plays an important role in recognizing the world. When we grasp an object, its size, color, material, position, and other properties are recognized. For example, the relationship between size, material, and weight are trained by the experience of manipulation.

Fig. 7.1

The brain receives information from the sensors, but each sensor has a different delay, and different regions of the brain received their signals, so the brain has to integrate the information. Perception differs from real sensory information, because the brain compensates for the lack of information

7.3 The System of Sense

Auditory sensors capture vibrations of the air. Visual sensors detect light. Molecules are detected by sensory cells in the nose and mouth, allowing us to smell and taste. Temperature, pain, and pressure are sensed on the skin by receptors such as Pacinian corpuscles, Ruffini’s corpuscles, and Meissner’s corpuscles. Sources of sensed sound and light can be distant, but smell comes from nearby. Arm length dictates distance for touch sensation, and the hand is an articulating sensor for capturing shape and hardness. Furthermore, there are other sensors that receive information passively.

When vibrations of the eardrum are passed on through the middle ear to the cochlea, sound is perceived. However, physical vibrations do not necessarily match the perception of the sound. The brain can pick out information from sensor signals (Fig. 7.1). For example, we can focus on a particular conversation in a noisy room. This phenomenon is explained by selective attention. This means that when we receive a sensation, not only bottom-up sensory signals, but also top-down signals affect perception. The “McGurk Effect” [2] is a good example. When subjects hear the syllables /ba-ba/ while viewing the lip motion /ga-ga/, most subjects report that the sound is /da-da/. This example also demonstrates that perception is not directly connected to sensor signals.

Another example is pseudo-haptics [3], which is a technique for simulating haptic sensation using visual feedback. This phenomenon is caused by the difference between the user’s displacement of the input device and the visual displacement of the object on the screen. This difference is known as the control/display (C/D) ratio (Eq. 7.1) in the human interface field.

$$\begin{aligned} \text{ C/D } \text{ ratio } = \frac{x_{hand}}{x_{display}} \end{aligned}$$

(7.1)

Fig. 7.2

A computer mouse measures hand motion on the desk. Leftward and rightward motion is reflected on the screen directly, but forward and backward motion is reflected indirectly as upward and downward motion. Still, we do not usually perceive this translation

One of the most popular human interfaces is the computer mouse. When we use a mouse, hand motion on the desk and cursor motion on the screen are not the same (Fig. 7.2). Forward and backward hand motions are translated respectively to upward and downward motions on the screen. We do not typically notice this translation. However, if the motion is slower or faster than expected, it becomes bothersome. So we can adjust the speed of the cursor through the computer’s settings. In Fig. 7.3, k is a parameter for position change. This is the same as the C/D ratio. Even though this ratio is not adjusted manually, we learn the relationship between our hand’s motion and the cursor’s motion. After learning or adjusting the C/D ratio, we cease to be aware of the transformation. Yet if this ratio is changed unexpectedly, we feel force. This phenomenon is the basis of pseudo-haptics.

Fig. 7.3

Input and Output signals of computer mouse

Pseudo-haptic feedback of isometric input devices was also tested using virtual springs [4]. Spring force (F) is defined by the spring constant (K) and displacement (x) as:

$$\begin{aligned} F&= K x \nonumber \\ x_{cursor}&= F_{hand}/K_{virtual} \end{aligned}$$

(7.2)

When cursor motion ($x_{cursor}$) is controlled by force ($F_{hand}$), the spring constant ($K_{virtual}$) is a parameter to adjust the motion.

$$\begin{aligned} K_{virtual}= F_{hand}/x_{cursor} \end{aligned}$$

(7.3)

The displacement (x) is inversely proportional to the spring constant (K), and $K_{virtual}$ is similar to the C/D ratio in Eq. 7.1.

The hand is pulled by the equilibrium position $x_{eq}$ with spring K and damper B (Fig. 7.4).

Fig. 7.4

Musculoskeletal system

Parameters K and B can also be defined as stiffness and viscosity , respectively. Along with equilibrium position , they are controlled by muscle contraction. The implications of these parameters are explained in later sections, but different combinations of these parameters can result in the same hand position. Conversely, C/D ratio is one parameter without redundancy. In this chapter, new human interface concept is proposed using these parameters.

7.4 Musculoskeletal Model

Hand force is produced by muscle tension . Skeletal muscle generates force in the pulling direction. For this reason, each joint has one or more pairs of muscles. The musculoskeletal model calculates joint torque from each muscle activation pattern.

7.4.1 Muscle Tension

The force that a muscle exerts depends on the muscle length and contractile velocity. Muscles have spring-like properties, with muscle force increasing as length increases. But when the length exceeds a threshold, force gradually decreases. Still, in daily life, muscle length does not typically go beyond this threshold, and also muscle tendons help to prevent this. Conversely, muscle force decreases as contractile velocity increases. But when muscle length increases, force is not affected by the velocity.

Each muscle is connected at a joint, and muscle length changes with the joint angle. For example, as the angle of the elbow joint increases, the length of the flexor muscle shortens while the opposing extensor muscle lengthens. This equates to a decrease in flexor tension and an increase in extensor tension.

7.4.2 Joint Torque

Net joint torque is calculated from the difference in flexor and extensor muscle tensions . This also means that each muscle tension is not directly related to joint torque.

Consider two examples, one where flexor muscle tension is 10 (arbitrary units) and extensor muscle tension is 7, resulting in a joint torque of 3, and another example where flexor muscle tension is 4, and extensor muscle tension is 1, again resulting in a joint torque of 3. What is the difference between these examples?

Fig. 7.5

Co-contraction

In Fig. 7.5, two conditions for muscle tension are plotted. When flexor and extensor muscle tensions are equal, joint torque becomes zero, and the arm is stable at some position (command set 1). At another strong contraction level, the same posture (command set 2) can also be obtained. We call this position equilibrium posture.

Fig. 7.6

Force generation

Fig. 7.7

Stiffness

7.4.3 Joint Stiffness

When joint position moves with the same contraction level, restoring force is generated (Fig. 7.6).

The magnitude of this force depends on each muscle’s contraction level. At the same posture, stiffness depends on muscle activation level (Fig. 7.7). The ratio of this restoring force and displacement is called joint stiffness.

$$\begin{aligned} Joint stiffness = \frac{restoring force}{displacement} \end{aligned}$$

(7.4)

In Fig. 7.5, restoring force is equal, but displacement is different. In Fig. 7.7, displacement is equal, but restoring force is different. For both cases, joint stiffness of command set 1 is greater than that of command set 2.

7.4.4 Equilibrium Position Change

To produce a trajectory, it has been hypothesized that the CNS sends only final posture information, and muscle activities are generated such that the joint is stabilized at the equilibrium position [5] (Fig. 7.8).

To test this hypothesis, Bizzi and colleagues performed an experiment using a deafferent monkey [6]. If the hypothesis were correct, the hand position would move to the end point just before the motion. The hand would be there because the brain (in this hypothesis) sends commands to stabilize the hand at the end position. However, when they did the experiment, they obtained different results. The hand moved to the middle position and then to the end point. They concluded that the brain sends commands to gradually shift the hand to the end point. So it came to be thought that the brain plans trajectory. However, if the brain can estimate the sensor signals using the feedforward model of our arm, in which the efference copy is an input signal, the experimental result can be explained using a simple end-point control hypothesis.

Fig. 7.8

Equilibrium control

7.4.5 Uniform Control Hypothesis

As mentioned in the previous section, we can change equilibrium position and stiffness independently. This means that we have a redundant control system. For example, in order to hold an object of weight m in hand, where the distance between the wrist joint and object position is d, and force F equals mg, the wrist must produce a torque of $mgd \cos \theta$ to compensate for the weight of the object. The brain changes the equilibrium position and stiffness to produce torque $\tau = K (\theta _{eq} - \theta )$. If the object’s weight is known, it is easy to set the equilibrium position and stiffness. For familiar objects, stiffness is set to an appropriately low value, and the brain changes the equilibrium position depending on the object’s weight. If the weight of the object is unknown, stiffness is controlled to the high value, and the difference between the equilibrium position and current position is small. High stiffness allows the hand to be stabilized even if the weight estimation is not completely accurate (Fig. 7.9).

Fig. 7.9

Uniform control for motion and force

Fig. 7.10

Experimental setup

7.4.6 Pseudo-haptic in 3D

The characteristics of pseudo-haptics have been tested for 2D motion on a frontal parallel screen. For virtual reality, binocular disparity is utilized by three dimensional displays. To determine the applicability of pseudo-haptics in virtual reality, the performance of pseudo-haptics in 3D space needed to be tested [7] (Fig. 7.10).

Results showed that the magnitude of pseudo-haptic sensation was reduced in the depth direction, compared to planar directions; i.e., discrimination of motion in the depth direction was decreased. This indicated that motion detection is important in perceiving pseudo-haptic sensation.

7.5 Pseudo-haptics by Stiffness

In Eq. 7.2, the spring constant is the ratio between force of hand and displacement on the screen, and it is similar to the C/D ratio. The stiffness of the musculoskeletal model can be estimated from EMG signals in real time and used to change weight perception . Co-contraction level can be modulated voluntarily, but it can also be non-voluntarily altered by environmental conditions. When we catch an object, stiffness level is dependent on the weight of the object [8]. Stiffness is low before contact, and increases just before contact onset. The maximum value of the stiffness is proportional to the weight of the object. If stiffness is related to weight perception, weight judgment would be different just before or after contact. To verify this hypothesis, we investigated the effect of temporal factors on weight perception [9]. We conducted ball-catching experiments in a virtual environment where the timing of load force exertion was shifted away from the visual contact timing (Fig. 7.11).

The perception of an object’s heaviness is not only dependent on its weight. It is well known that spatial information of an object, such as size, can easily deceive our perception of its heaviness. To further understand neural mechanism underlying weight perception, we investigated effects of temporal information on the weight perception. We conducted experiments in which a falling ball is displayed on a screen and load force of the ball is exerted on the hand by a haptic device. By shifting the timing of load force exertion away from visual contact timing (i.e., time when the ball hit the hand in the display), we found that the ball was perceived heavier/lighter when force was applied earlier/later than visual contact. We also found that the illusion in perceived heaviness induced by the time offset between visual and haptic contact timing became smaller after participants had been conditioned to the time offset. These results suggest that the illusion found in our experiments was not caused by the physical time offset between force exertion and visual contact but by the perceived time offset between them and/or estimation error in force exertion timing.

Fig. 7.11

Experiment system. A virtual red ball (radius = 2 cm) and a black square cursor (width = 10 cm, height = 2 cm) projecting the hand position in the vertical direction were displayed on a plasma display. Subjects held a ball-shaped plastic grip attached to a SPIDAR haptic device, which consisted of eight motors and strings. Load force was applied through tension in the strings by the motors. The grip position was calculated from angle encoders attached to the motors. Subjects wore a pair of noise-canceling headphones to reduce the sound generated by the motors of the haptic device

7.5.1 Experimental Protocol

Six right-handed male adults (age: 21–39) took part in three experiments 1, 2, and 3, conducted on separate days. Experiment 1 was conducted to investigate how the time offset between load force exertion and visual contact affects perceived weight of the ball. In Experiments 2 and 3, we investigated how weight perception changes after participants were persistently exposed to constant time offsets. All participants performed Experiment 1 first. The order of Experiments 2 and 3 was randomized among all participants, with half performing Experiment 2 before 3 and the other half performing Experiment 2 after 3. Each experiment was organized into three sessions: “Conditioning”, “Simultaneity Test”, and “Weight Perception Test” sessions, presented in this order. Rest breaks of several minutes were taken between sessions.

7.5.2 Conditioning Session

Participants performed 80 ball-catching trials in the Conditioning session. The inter-trial interval was 2 s. The sequence of events in a single trial is shown in Fig. 7.12. At the beginning of the trial, the ball appeared at 80 cm above the blue bar, accompanied with a beep sound, and started falling after a random delay. The ball load force was then applied with or without time offset from visual contact.

Fig. 7.12

The sequence of events in a single ball-catching trial during the Conditioning session. The load force is applied at the same time (Experiment 1), 60 ms before (Experiment 2), or 60 ms after (Experiment 3) the ball contacts the hand cursor in the display

In Experiment 1, the load force was synchronized with visual contact. In Experiment 2, the force was applied 60 ms before visual contact, and in Experiment 3, 60 ms after visual contact. The magnitude of load force was 3.92 N for all Conditioning sessions. This value was selected to simulate the feeling of catching a ball weighing 400 g. In accordance with the time offsets used in the three experiments, we denoted the Conditioning sessions as “sync” (Experiment1), “lead” (Experiment2), and “lag” (Experiment3).

7.5.3 Simultaneity Test Session

Each participant underwent 99 ball-catching trials in the Simultaneity Test session. The sequence of the events in a single trial was the same as that shown in Fig. 7.12, except for the time offset values. The offset for each trial was selected randomly from the list. The magnitude and the duration of load force in each trial were 3.92 N and 1 s. Just as in the Conditioning session, we asked the participants to counteract the load force to catch the virtual ball. The participants were instructed to make judgments about the temporal order of visual contact and force exertion. We asked them to report which event occurred first by pressing the left or right button of a computer mouse held in their left hands.

We instructed participants to counteract the load force so as to keep the black cursor within the blue bar as consistently as possible.

Fig. 7.13

The sequence of events in a single weight judgment trial during the Weight Perception Test session. After applying the constant magnitude reference force, participants performed ball-catching. The timing of load force exertion was chosen randomly from three candidates corresponding to the times simultaneous to (0 ms offset trials), 60 ms before ($-60$ ms offset trials), or 60 ms after (+60 ms offset trials) visual contact of the ball

7.5.4 Weight Perception Test Session

In each of the three experiments, the Weight Perception Test session was organized into five sets. A single set was composed of 99 trials in Experiment 1, and 76 trials in Experiment 2 and 3. Rest breaks of several minutes were taken between sets. In each set, participants first performed 30 ball-catching trials without any perceptual judgment. The time offsets for the first 30 ball-catching trials were the same as those in the Conditioning session in each experiment. The rest of the trials were weight judgment trials in which participants were asked to compare the heaviness of the ball and a reference force. Figure 7.13 shows the time sequence of the events in a single weight judgment trial. At the beginning of the trial, the black cursor disappeared from the display, accompanied with a beep sound. After a random delay, a reference force with magnitude 3.92 N was applied for 1 s. After the 1 s time interval, the ball and the black cursor appeared with a second beep sound. Participants then performed the ball-catching task after a random delay ranging from 0.75 to 1.25 s. Time offset was again imposed between visual contact and load force exertion. Its value was selected randomly from one of the following values: $-60$, 0, or $+60$ ms in Experiment 1, $-60$ or 0 ms in Experiment 2, and 0 or $+60$ ms in Experiment 3. Here, negative or positive offset signs indicate that load force preceded or followed visual contact, respectively. Each offset appeared 23 times in every set. Magnitude of load force was also selected randomly. The magnitude values are listed in Table 7.2 with their appearance frequency for each time offset within a single set.

Participants were asked not to overcorrect for the reference force. The instructions for the ballcatching task were the same as those for the Conditioning session. After catching the ball, participants were required to judge the heaviness of the load force compared to the reference force. Participants reported which force they perceived as heavier by pressing the left or right button of a computer mouse.

After performing ball-catching trials with 60 ms advanced or delayed load force exertion, participants’ subjective judgment on the simultaneity of visual contact and force exertion changed, reflecting a shift in perception of time offset. In addition, timing of catching motion initiation relative to visual contact changed, reflecting a shift in estimation of force timing. We also found that participants began to perceive the ball as lighter after conditioning to 60 ms advanced offset and heavier after the 60 ms delayed offset. These results suggest that perceived heaviness depends not on the actual time offset between force exertion and visual contact but on the subjectively perceived time offset between them and/or estimation error in force timing.

Table 7.2 Load force values and frequencies in the Weight Perception Test session

7.5.5 Perceptual Judgment Analysis

For the Simultaneity Test session, the judgment of participants was modeled to a psychometric curve. The probability of judging “load force preceding visual contact” was fitted with a sigmoid function,

$$\begin{aligned} prob(force\ first) = \frac{1}{1 + \exp ( \theta _0 + \theta _1 \varDelta t)} \end{aligned}$$

(7.5)

where $\varDelta t$ is time offset, and $\theta _0$ and $\theta _1$ are the regression coefficients. A psychometric curve was made for each participant using their individual judgments. A group psychometric curve was also made from the judgments across all participants. The point of subjective simultaneity (PSS), where $\varDelta t$ gives $prob = 0.5$, can be calculated as PSS = $-\theta _0/\theta _1$.

For the Weight Perception Test session, the participants’ judgments were again modeled to a psychometric curve. The probability of judging that “the ball was heavier than the reference force” was fitted with a sigmoid function,

$$\begin{aligned} prob(ball\ heavier) = \frac{1}{1 + \exp ( \phi _0 + \phi _1 \varDelta F)} \end{aligned}$$

(7.6)

where $\varDelta F$ is the percent difference in load force magnitude compared to that of the reference force and takes a negative value when the load force is comparatively smaller. Both individual and group psychometric curves were computed for each time offset used in each of the experiments. The point of subjective equality (PSE), where $\varDelta F$ gives $prob = 0.5$, can be calculated as PSE $= -\varPhi _0 / \varPhi _1$. The PSE indicates the magnitude of load force perceived to be the same as that of the reference force.

Fig. 7.14

Group-average psychometric curves after the three types of time offset conditioning. Circles, squares, and triangles represent the group-average probability for the “sync,” “lead,” and “lag” conditioning, respectively

7.5.6 Result of Simultaneity

For the Simultaneity Test sessions, we analyzed how perception of the temporal order of the visual contact and load force events changed after three different types of conditioning. Psychometric curves for each participant and each conditioning type are shown in Fig. 7.14. The subjective simultaneity of the two events was evaluated by the PSS of the psychometric curves (see Eq. 7.6). The average PSS across participants were $-44.7$ (SD: 14.0) ms, $-4.5$ (SD: 4.0) ms, and 36.5 (SD: 13.1) ms after the “lead”, “sync”, and “lag” conditionings, respectively. The group-average psychometric curves are also shown in Fig. 7.14. The curves for “lead” and “lag” conditioning were clearly shifted leftward and rightward, respectively, in comparison to “sync” conditioning. The PSS shifts for “lead” conditioning with respect to “sync” conditioning were $-40.1$ (SD: 15.2) ms averaged across participants. This was significantly less than zero according to a one-sided t-test. On the other hand, the PSS shifts for “lag” conditioning with respect to “sync” conditioning were 40.8 (SD: 15.4) ms averaged across participants. This was significantly greater than zero. Therefore, PSS shifted toward the direction of persistently exposed time offset during the Conditioning session.

Fig. 7.15

Weight judgment. a Psychometric curves for each participants after “sync” conditioning. Line color represents time offsets in weight judgment trials. b Group-average psychometric curves after “sync” (solid lines), “lead” (dash-dotted lines), and “lag” (dotted lines) conditionings

7.5.7 Result of Weight Perception

After “sync” conditioning.

Figure 7.15a shows each participant’s psychometric curves of weight perception after “sync” conditioning. Although psychometric curves differed from participant to participant, they moved toward the right as offset increased from $-60$ ms to $+60$ ms. This tendency can be clearly seen in the plot of group-average psychometric curves (see solid lines in Fig. 7.15b). The psychometric curve shifts indicate that the same magnitude of load force was perceived differently as time offset changed. For example, when the load force magnitude was the same as that of the reference force ($\varDelta F = 0\,\%$), the probability that participants perceived the ball heavier became larger as the offset became negative (i.e., the load force preceded).

The difference in perceived heaviness can also be evaluated by the difference in PSE of the psychometric curves (see Eq. 7.6). The average PSE across participants were $-6.9 (SD: 7.2) \%$, $3.9 (SD: 7.4) \%$, and $14.3 (SD: 4.5) \%$ for $-60$, 0, $+60$ ms offset trials, respectively. Note that the smaller the PSE, the heavier the load force perceived. The average PSE shifts from 0 ms to $-60$ ms offset trials was $-10.7 (SD: 3.5) \%$ and was significantly less than zero. The average PSE shifts from 0 ms to $+60$ ms offset trials was $10.4 (SD: 9.7) \%$. This shift was significantly greater than zero. Therefore, load force exerted earlier than visual contact was perceived as heavier than that exerted at the same time as visual contact, and when the load force was exerted later than visual contact, it was perceived as lighter.

After “lead” and “lag” conditioning

Figure 7.15b shows group-average psychometric curves for the weight judgments in the three experiments. After “lead” conditioning, psychometric curves for both of $-60$ ms and 0 time offsets shifted toward the right compared for “sync” conditioning. The rightward shifts indicate that the participants perceived the ball’s weight to be lighter. On the contrary, after “lad” conditioning, the curves for both of 0 ms and $+60$ ms time offsets shifted toward the left compared for “sync” conditioning. The leftward shifts indicate that the participants perceived the ball’s weight to be heavier. Change in perceived weight was evaluated by PSE shifts averaged across participants. The average PSE shifts from “sync” to “lead” conditionings were $6.2 (SD: 5.3) \%$ and $4.1 (SD: 9.4) \%$ for $-60$ ms offset trials and 0 ms offset trials, respectively. The PSE shift for $-60$ms offset trials was significantly greater than zero ($t(5) = 2.84$; $P = 0.018$). Although the PSE shift for 0ms offset trials was not significantly greater than zero, it tended to shift rightward. The average PSE shifts from “sync” to “lag” conditionings were $-6.2 (SD: 3.0) \%$ and $-5.2 (SD: 3.9) \%$, for 0 ms and $+60$ ms offset trials, respectively. Those shifts were significantly less than zero. These results indicate that the participants’ ball weight perception was changed by being conditioned to time offset between load force exertion and visual contact. The participants began to perceive the ball’s weight as lighter after “lead” conditioning and heavier after “lag” conditioning.

Fig. 7.16

Weight judgment results

We found that subjects weight perception changed after conditioning to time offset (Fig. 7.16). The weight was perceived lighter after conditioning to the $-60$ ms time offset, and perceived heavier after conditioning to the $-60$ ms time offset. Therefore, time offset itself was not the causal factor for the weight illusion.

7.5.8 Relationship Between PSE and PSS

The results of the weight judgment trials in Experiment 1 revealed that the weight of the falling ball was perceived differently by introducing time offsets between load force exertion and visual contact (Fig. 7.15). In Experiment 2 and 3, we also found that the weight of the ball was perceived differently after “lead” and “lag” conditioning, even though the time offsets were the same as those used in the weight judgment trials in Experiment 1 (Fig. 7.15). Therefore, the perceived weight illusion observed in Experiment 1 seems not to be related to actual physical time offset between visual contact and load force exertion. Rather, the illusion in weight perception seems to be connected to the participants’ subjective perception of time offset. This subjective time offset is thought to be modified by shifts in PSS after “lead” and “lag” conditioning. To show how perceived weight is related to physical or subjective time offset, the PSEs of group-average psychometric curves shown in Fig. 7.15 are plotted with respect to their corresponding physical or subjective offsets (Fig. 7.17). Here the subjective time offsets were calculated by subtracting the PSS of the group-average psychometric curve in each experiment from the physical time offsets used in the Weight Perception Test session in each experiment. We can see that the same physical offset yielded different PSE values (black open marks). On the other hand, PSE values plotted with respect to subjective offset (gray filled marks) increased approximately linearly with subjective offset value. The correlation coefficient of PSS and subjective time lag was 0.98.

Fig. 7.17

PSEs of group-average psychometric curves for weight judgments in all experiments are plotted with respect to actual (physical) time offsets as black unfilled marks. PSEs are also plotted as gray filled marks with respect to subjective time offsets (physical offset minus PSS of group-average psychometric curves for temporal simultaneity). Shapes of the marks represent the Conditioning sessions. The gray solid line is a linear regression of the gray marks

7.5.9 Relationship Between PSE and Estimation of Error in Force Exertion Timing

In addition to the subjective perception of time offset, the estimation of force exertion timing in sensorimotor system also seems to be modified after “lead” and “lag” conditioning. Here we analyze the relationship between perceived weight and estimation error in force exertion timing. Although we cannot directly measure the estimation of force exertion timing, we can infer it from the motion initiation timing relative to visual contact. Let us assume that ball-catching motions are initiated some fixed second in advance of the estimated timing of force exertion. This assumption is supported by other experiments in which the timings of muscle activity and catching motion were found to be consistently initiated a few hundred millisecond before the ball contacts the hand [10–14]. We also assume that the margin between the motion initiation timing and the estimated timing varied among participants, but did not change within a single participant in the three experiments. According to these assumptions, we inferred changes in the estimation of force timing by analyzing changes in the motion initiation timing relative to visual contact. Figure 7.18 plots the PSEs of group-average psychometric curves shown in Fig. 7.15 against the estimation error in load force exertion timing. Note that we assumed that there was no estimation error in 0 ms offset trials after “sync” conditioning (Experiment 1). The estimation error was then calculated by subtracting group-average motion initiation timing in each experiment from the sum of time offset and group-average motion initiation timing in Experiment 1. For example, the estimation error in $+60$ ms offset trials in Experiment 3 was 20.1 ($=60 + (-75.3) - (-35.9)$), where the values of $-75.3$ and $-35.9$ ms correspond to group-average motion initiation timings in Experiment 1 and 3, respectively (values of group-average motion initiation timing in each experiment is described in Sect. 7.2). We can see that the PSE values increased almost linearly with the estimation error. The correlation coefficient of PSE and the estimation error was 0.99.

Fig. 7.18

PSEs of group-average psychometric curves for weight judgments in all experiments are plotted with respect to the estimation error in load force exertion timing. The estimation error for $\alpha$ ms offset trials in Experiment $\beta$ was calculated by subtracting group-average motion initiation timing in Experiment $\beta$ from the sum of $\alpha$ ms offset and group-average motion initiation timing in Experiment1

7.6 Representation of Motion in the Brain

Stiffness is closely related to weight perception (described in Sect. 7.5.9). But where is stiffness information represented in the brain? Muscle activations are represented in the primary motor cortex. But stiffness of the hand is related to either joint coordinates or environmental coordinates. Understanding the coordinate systems employed by the brain is crucial to uncovering the mechanisms of weight perception.

The brain allows skillful manipulation of the body to interact with the external environment. This sophisticated and flexible operation involves transformations between coordinate systems of the internal body and external environment, possibly computed in distributed brain regions. The internal coordinate system is body- and/or joint-centered, and may thus be represented intrinsically, whereas the external coordinate system refers to points outside the body.

Fig. 7.19

a Schematic of participant postures and relation between movement directions and tasks according to visual instructions in three different wrist postures. b Block design for the fMRI experiment

The representation of intrinsic (i.e., joint) and extrinsic (i.e., movement) coordinate frames were analyzed using functional magnetic resonance imaging (fMRI). During fMRI acquisition, healthy human participants performed isometric flexion and extension tasks (Fig. 7.19) in different forearm postures. In a pronated posture (Pro), an upward force exertion was equivalent to an extension task and cued with an up arrow (Up/Ext), whereas a downward force exertion was equivalent to a flexion task and cued with a down arrow (Down/Flex) (Fig. 7.19a, left panel). In a supinated posture (Sup), an upward force exertion was equivalent to a flexion task and cued with an up arrow (Up/Flex), whereas a downward force exertion was equivalent to an extension task and cued with a down arrow (Down/Ext (Fig. 7.19a, center panel).

In a midway posture between Pro and Sup (Mid), a leftward force exertion was equivalent to a flexion task and cued with a left arrow (Left/Flex), whereas a rightward force exertion was equivalent to an extension task and cued with a right arrow (Right/Ext) (Fig. 7.19a, right panel).

Figure 7.19b shows the block design for the fMRI experiment. Execution tasks (Flex and Ext) were instructed with up and down arrows (in Pro and Sup) or left and right arrows (in Mid) inside a gray box, and a no-force task (Still) was instructed with a vertical bar (in Pro and Sup) or a horizontal bar (in Mid).

Fig. 7.20

a Accuracy for Flex versus Ext classification and up versus down classification using Pro and Sup data. b Classification accuracy using Mid data

Figure 7.20a shows an accuracies for Flex versus Ext classification and Up versus Down classification using Pro and Sup data. Gray bars represent mean accuracies of 10 participants (Participant 1–10) calculated using 6-fold cross validation and black bars represent grand mean accuracies averaged across participants. Error bars denote standard deviation. There was no significant difference between the grand mean accuracies of the two classifications. The obtained classifiers were considered direction-specific (left group; Up vs. Down classification) and join-specific (right group; Flex vs. Ext classification) classifiers. Figure 7.20b shows the classification accuracies using Mid data. Gray bars are accuracies of 10 participants (Participant 1–10) and black bars are grand mean accuracies averaged across participants. The trained classifier did not specifically discriminate joint action (Flex vs. Ext) or movement direction (Left vs. Right), but discriminated mixed joint action and movement direction features (Left/Flex vs. Right/Ext) since only Mid data was used for classifier training. Results using direction-specific (Up vs. Down) classifiers for each participant (gray bars) and grand mean accuracy averaged across participants (a black bar). The direction-specific classifier for each participant was the mean of 6 classifiers obtained from 6-fold cross validation using Pro and Sup data (Fig. 7.20a, left group). Results using joint-specific (Flex vs. Ext) classifiers for each participant (gray bars) and grand mean accuracy averaged across participants (a black bar). The joint-specific classifier for each participant was the mean of 6 classifiers obtained from 6-fold cross validation using Pro and Sup data (Fig. 7.20a, right group).

Fig. 7.21

Anatomical mapping of statistical group analysis results

Mean contribution ratios of intrinsic and extrinsic coordination in four ROIs, primary motor cortex hand knob (M1), ventral premotor area (PMv), dorsal premotor area (PMd), and supplementary motor area (SMA), were calculated and averaged across participants.

The motor-related cortical regions identified in the current study match those of prior studies in this field. More specifically, our results revealed that intrinsic coordination was mainly associated with M1, while extrinsic coordination was mainly associated with PMv, PMd, and SMA. This is reasonable, considering the layered organization of the cerebral cortex and signal pathways to M1, PM and SMA. Since M1 transmits output signals to the muscles through a number of layers, the selection of flexion and extension tasks, which is associated with only internal body control, may be represented in M1. At the same time, the other layers of M1 receive input from PM, SMA, and cingulate motor area, which may be the reason why direction of movement is encoded not only in PM regions but also in M1 [15, 16]. This in turn would explain why, in our analysis, 39 % of M1 voxels showed contribution to extrinsic coordination in Fig. 7.21. Prior studies have shown that PMv represents direction of action [15–17], but others suggest that premotor areas operate at a hierarchical level comparable to M1 since they appear to have direct connections with spinal motoneurons, particularly those innervating hand muscles [1, 18]. In this respect, our results replicated these findings, and we can explain why statistically significant clusters were not found in the PMv-ROI while a relatively high amount of PMv voxels (i.e., 42 %) contributed to intrinsic coordination (Fig. 7.21). PMv is considered to be involved in visuomotor transformations [19, 20], and we believe our results support this view. Tasks in our experiment were cued with graphical arrows. Thus, visuomotor transformations from direction information in the external world to action information in the internal body would be necessary. In addition, a path of visuomotor transformations required for grasping connects the dorsal extrastriate cortex and PMv via the anterior intraparietal area [21]. Although our experimental tasks were not finger movements, similar information may be processed for wrist flexion and extension. PMv then outputs visuomotor transformed information to M1, which may also explain the 42 % contribution by PMv to intrinsic coordination. Rizzolatti and colleagues also showed that a path of visuomotor transformations required for reaching connects the parieto-occipital extrastriate area and PMd separately from the path required for grasping [21]. Since, in the current study, PMd showed a 70 % contribution to extrinsic coordination, reaching and wrist flexion and extension may also share similar information. Other studies have also revealed that PMd is associated with motor planning and initiation [22, 23] and action prediction [24]. Considering the experimental design of the current study, participants performed tasks according to periodic visual stimuli and had to memorize the relationship between each task and respective cue. Therefore, participants might have naturally predicted and prepared for the next task, which would attribute to PMd activation. Therefore, these brain activities might be necessary for both joint action and movement direction.

7.7 Applications of a Sense-Centered Human Interface

7.7.1 TCieX

We developed a sense-centered human interface named the Touch-Centric Interaction Embodiment Exploratorium (TCieX) [25 ] . This system uses visual interaction to communicate weight, stiffness, or viscosity by exploiting pseudo-haptics in human-computer interaction. TCieX is a collection of simple interaction test suites that allow experience of different combinations of multimodal interactions. This system measures the hand motion and pressure on a display and biological signals (e.g. EMG signals) simultaneously to control cursor position, velocity, and acceleration (Fig. 7.22).

An illusory sense of haptics also occurs with the combination of these signals.

Fig. 7.22

TCieX by iPad$^{\textregistered }$

7.7.2 Surgery Robot

The da Vinci Surgical System$^{\textregistered }$ (Intuitive Surgical, Sunnyvale, CA) is one of the most well-known robot surgery systems . A 3D high definition vision system provides a clear and magnified view for the surgeon. The surgeon manipulates a master robot to intuitively control a slave robot. The master-slave robot system translates hand motion into smaller, more precise motion without haptic information. It is technically feasible to add haptic sensation to the slave robot arm, but difficult to add force-torque sensors because the tip of the slave robot is disposable to maintain sterility. Force feedback is useful to prevent user fatigue. For this reason, force feedback without force sensors has been developed [26, 27].

Fig. 7.23

Schematic of external force estimation by biological signals

Figure 7.23 shows a schematic for estimation of external force. The operator can see the motion of the slave arm, and the stiffness of the slave is constant ($K_{slave}$).

$$\begin{aligned} F_{hand} = \delta x_{slave} \times K_{slave} \end{aligned}$$

(7.7)

This allows the operator to estimate the force of the slave robot. Our sense-centered human interface provides an alternative means of acquiring force sensation [28].

7.7.3 Power-Assist Robot

The conventional power-assist robot is controlled by torque, which is estimated from EMG signals. The power is proportional to the contraction level. However, this kind system is liable to become unstable particularly during posture control. Our new control method based on equilibrium position (described in Sect. 7.4.3) offers greater stability than conventional methods.

When holding an object, hand position is kept at the same position. The co-contraction level can also change while keeping the same posture. This means that the command scheme has redundancy, and another parameter exists for control. Our arm can be modeled as a mass-spring-damper system using muscle spring-like properties. Hand force can be defined as $Force = K_{spring constant} \times (x_0 - x) - B_{viscosity} \dot{x}$, where $x_{0}$ is the estimated position from EMG activity and represent equilibrium position. Values x and $\dot{x}$ are current position and velocity measured by the position sensor of the motor. Figure 7.24 shows a power-assist robot with position and impedance controlled by EMG of the forearm.

Fig. 7.24

Power-assist robot using equilibrium-based control

Fig. 7.25

Equilibrium-based control is more stable than the conventional torque-based control

The conventional method of control uses joint torque estimated from EMG. During posture control, stabilization of posture is achieved by exerting joint torque. Power-assist systems enhance this exertion, and this causes positional errors, which can be observed as oscillatory motion. In our system, joint torque is exerted toward the equilibrium position [14] (Fig. 7.25).

7.8 Conclusion

Sense-centered human interfaces have the potential to influence perception by exploiting the intricacies of the brain’s sensory systems. Yet further development of body and brain models, sensor devices, and signal processing techniques, as well as more quantitative evaluations of the perception is needed.

The techniques described here offer a framework for new human-interfaces as well as new humanoid robots capable of detecting sensation as humans do, allowing them to work in environments meant for humans.

Figrue 7.26 shows a matrix map of virtual reality techniques from the standpoint of force feedback. The horizontal axis indicates visual information, real or virtual. The vertical axis indicates force feedback. Virtual reality (VR) replicates the real world by providing sensory and visual feedback using computer-simulated devices.

Fig. 7.26

Matrix map of virtual reality from the standpoint of force feedback

Tele-operation is a prime example of a VR system. Visual information from a camera is real, but we cannot touch what is displayed. So haptic devices reproduce force feedback, which is measured by a force-torque sensor. VR allows for the replication of a broad range of real-world sensations in a virtual world.

Pseudo-haptics is a technique for simulating haptic sensation using visual feedback without a haptic device. Our sense-centered human interface also provides haptic sensation without haptic device, but the difference between them is the type of input information. Pseudo-haptics uses input information which corresponds to the output signal. C/D ratio is a ratio of output to input. Our system also uses sensor signals, but the equilibrium position or stiffness does not directly correspond to the output information. Also these types of information are typically controlled unconsciously.

When we push a button, the force used depends on the person. The number of possible input key combinations of a 4-digit passcode is $10^4$. If two different levels of force are added to the system, the number of possible combinations increases to $20^4$ [29]. Applied force for pushing buttons is usually controlled unconsciously, is consistent over time, and differs between people, making it potentially useful in security applications. Force sensors are effective in measuring force for human interface applications. But EMG signals reflect not only force information, but also stiffness and equilibrium position. Sense-centered human interfaces based on EMG can be used to estimate subjective perception and enhance sensation.