Keywords

1 Normal Motor Function Needs Somatosensation

Motor neural prostheses are assistive devices that aim to restore normal movements for persons with injury or disease (Lebedev and Nicolelis 2006; Velliste et al. 2008). These systems harness neural activity to control motor prostheses with the promise of providing natural and effortless mobility with a degree of fluidity that could someday approach that of unimpaired movement. Different motor impairments spare varying degrees of residual function (compare, for example, spinal-cord injury (SCI) to the amputation of a limb), meaning that the most appropriate approach to artificial sensory feedback will be unique in each case. As a result, there is a wide range of neural prosthetic solutions that have been proposed and developed for restoring movements, extending from myoelectric signals extracted from muscles and nerves (Parker and Scott 1986; Kuiken et al. 2009) to direct cortical control (Hochberg et al. 2006, 2012). Despite their diversity, all of these approaches share a common need: somatosensory feedback (Lebedev and Nicolelis 2006; Hatsopoulos and Donoghue 2009; Lebedev et al. 2011; Gilja et al. 2011; Weber et al. 2012).

Somatic sensations (those that originate from the limbs and body, chiefly touch and proprioception) are essential for fine control of movements and the dexterous manipulation of objects (Johansson and Flanagan 2009). They also reduce the effort required to make goal-directed movements while improving their reliability. More specifically, somatosensation relieves the need for constant visual attention to manipulated objects and permits movements made outside of the visual field (into pockets, etc.). It also enables tasks that are difficult to accomplish even with full and direct vision. This critical aspect can perhaps be appreciated by considering the motor impairments exhibited by persons suffering from sensory deafferentation, impairments that persist despite a fully intact motor pathway. For example, the seemingly simple task of lifting an object (Johansson and Westling 1984) or striking a match is more challenging and takes longer to accomplish when attempted without cutaneous sensation from the fingers. Furthermore, without proprioception it is difficult to make spatially precise movement trajectories that require multi-joint coordination (Sainburg et al. 1993). Finally, despite the fact that vision is often viewed as the dominant sensory modality for humans, somatosensation actually offers more precision along some spatial axes (van Beers et al. 2002) and has a shorter feedback latency than vision (Omrani et al. 2013).

As one would predict from its importance for normal limb function, there is growing direct evidence that artificial proprioception would improve motor performance with prosthetic limbs. Monkeys making virtual reaches with a Brain-Computer Interface (BCI) controlled cursor demonstrated better performance levels when an exoskeletal robot passively moved their forelimb to track the cursor movement, compared to performance when their limb was held still or moved randomly (Suminski et al. 2010). Although visual feedback of the state of the cursor was available in all conditions, BCI control only approached natural levels of performance and fluidity when additional state information was provided by intact proprioceptive feedback. A similar result was observed when decoding movement intention from humans with intact proprioception (Gomez-Rodriguez et al. 2011), where closing the sensorimotor feedback loop improved the user’s degree of control over the BCI. We expect that the benefit of proprioceptive feedback on BCI control will persist, even as the BCI control algorithms continue to improve (Gilja et al. 2012; Orsborn et al. 2012)—just as deafferented patients with nominally intact motor systems exhibit motor impairments, a motor neural prosthesis without somatic sensation will remain functionally impaired.

To optimize BCI control, an artificial sensory signal must fulfill the functions of natural proprioception. In particular, we will consider two criteria that we hypothesize will be sufficient for improving BCI control. Namely, we want to design an artificial signal that can provide enough information to be used alone when vision is not available (sensory substitution), and that can be integrated with vision to improve motor performance when both inputs are available (sensory augmentation).

2 Approaches for Artificial Somatosensation

Given the importance of somatosensory feedback for motor neural prosthetic systems, we next ask how such feedback should be provided. As with the selection of a motor decode strategy, the most suitable route for somatosensory feedback will depend on the specifics of the injury and the preferences of the user. For example, an amputee with an upper-limb prosthetic arm may be best served by stimulation of residual peripheral nerve afferents (Schiefer et al. 2010). In contrast, for SCI patients, the peripheral route may not be a viable option and a more central site of stimulation would be required.

In the following, we focus on the problem of restoring somatic sensation for complete SCI or other conditions that result in functionally complete somatosensory loss below the neck. While these strategies may also apply in situations with less severe disability, we choose to focus SCI because there are fewer existing and viable strategies for restoring somatosensation to these individuals. SCI imposes strong restrictions on the design of neural prosthetic devices. Motor commands are only available above the lesion site, so in many cases they must be read out from the brain. Similarly, sensory afferents from below the lesion are few or non-existent. While both invasive and non-invasive devices could be used in this case, we argue next that an invasive approach is preferable for both neural read-out and sensory write-in.

2.1 Non-invasive Approaches

There are several non-invasive ways that a motor command signal could be obtained in SCI, including measurement of eye, head, face, or neck movement—directly or via electromyogram (EMG) or electrooculogram (EOG) recordings—or with external recording of neural signals from the brain (electroencephalogram, EEG). These controllers can be classified as either “direct” or “indirect.” With indirect control schemes, users learn to control the prosthetic device with a substitute effector. Clinically successful examples of indirect control include sip-and-puff systems, head-mounted wands and joysticks, and voice commands. These indirect approaches are simpler and less expensive than other methods, but their information bandwidth is limited. In contrast, direct control schemes measure the control signal from EEG recordings, although eye position control of a computer cursor could also be included. Notably, an EEG-based system has recently achieved information bandwidths comparable to invasive BCIs (Bin et al. 2011). However, these information rates were obtained by analyzing the temporal pattern of visual evoked potentials (VEP) during the serial presentation of a large set of discrete stimuli. The system was able to estimate which of a set of targets was being visually fixated at a rate of approximately every 2 s; significant advances would be needed to use this approach to obtain smooth, continuous control of a prosthetic device.

In principle, restoration of somatosensory feedback via non-invasive BCIs could also include direct and indirect approaches, however in the case of SCI, where the sensory periphery is unavailable, there are currently no viable non-invasive technologies for stimulating the somatosensory pathways of the limb. This leaves only indirect approaches, or  “sensory substitution”, where sensory information intended for one modality is translated into another. Sensory substitution has employed tactile, auditory, and superficial electrical stimulation to replace systems like natural vision (Nau et al. 2013) and vestibular function (Vuillerme et al. 2011). While non-invasive systems have clear appeal, tactile stimulation can be uncomfortable, and a device of suitably high bandwidth placed on the head or neck is likely to be bulky or awkward. We argue below that a fully implantable invasive device may, in the end, be more tolerable to many patients.

Despite potential limitations for rehabilitation, studies of sensory substitution provide key insight into the central role that sensorimotor learning can play in systems for artificial somatosensation. In a seminal sensory substitution experiment, blind subjects were taught to detect visual objects using a system that translated video input from a camera into a matrix of tactile inputs on the subject’s back (Bach-y-Rita et al. 1969; Bach-y-Rita and Kercel 2003). Users of this system had great success, even learning to recognize faces and partially occluded objects, and perceived “the external localization of stimuli,” meaning that the objects they sensed seemed to come directly from the camera rather than having to be interpreted from a sense of tingling or touch on their backs (Bach-y-Rita et al. 1969; Bach-y-Rita and Kercel 2002). Importantly, subjects only learned to distinguish objects when they were able to actively manipulate the movement and perspective of the camera, observing the changes in feedback as a result of their actions. If instead the camera was stationary or moved by another person, subjects did not “learn to see,” and instead had to interpret the signals in terms of touch and vibration. We conclude that active exploration is essential for learning of new modalities, but once learned, new modalities can be directly perceived, without the need for a translation step. We believe that these principles can usefully inform invasive approaches for sensory feedback.

2.2 Invasive Approaches

The key challenge for invasive approaches to BCI is the long-term stability and safety of the neural interface. Historically, implanted devices have suffered from mechanical failures (Barrese et al. 2013), neurovascular damage (Kozai et al. 2010), immune response (Woolley et al. 2013; Polikov et al. 2005),  infection and other difficulties associated with percutaneous implants. However, recent initiatives aim to solve these problems by improving device design (smaller, more flexible electrodes), taking new approaches to avoid signal degradation (ECOG/LFP-based decoding as well as using robust multi-unit decoding; Schalk et al. 2008; Ledochowitsch et al. 2013; Chestek et al. 2011), and fully-implantable low-power wireless recording systems (Alam et al. 2013) that may alleviate both the medical risks and aesthetic concerns of percutaneous devices. These advances, and others, in the field of neurotechnology suggest that safe and reliable invasive neural interfaces are on the horizon.

A major advantage of using invasive systems is the potential for high-bandwidth readout, due to their high spatial resolution, frequency range, and robustness to noise. Invasive systems now achieve information transfer rates of about 2 bits/s (Mulliken et al. 2008) in the continuous control of cursors and robotic arms, in large part due to improved algorithmic design (Orsborn et al. 2012; Gilja et al. 2012; Li et al. 2011; Shanechi et al. 2013), and continued progress is expected. Still, BCIs have yet to approach the performance levels of natural human movement (4.5 bits per movement, for movements that should take much less than 1 s to perform; Georgopoulos and Massey 1988). We attribute much of the remaining shortcoming in BCI control to the need for somatosensory feedback.

2.3 Biomimetic Encoding of Somatosensory Information

Invasive neural interfaces enable not only high-bandwidth output from the brain, but also multi-channel input into the brain. Electrical stimulation has been used to drive targeted manipulations of neural activity, inducing artificial somatosensation or perturbing natural somatosensation, in both animal models and human (e.g., Penfield and Boldrey 1937; Butovas and Schwarz 2003; Tabot et al. 2013). As the number and density of stimulating electrodes increases, so does the potential complexity of the spatiotemporal pattern of induced activity. This opportunity brings with it the challenge of choosing the right spatiotemporal activity patterns and determining how to create those patterns via stimulation. The dominant strategy so far has been biomimetic (Fagg et al. 2007)—trying to replicate the patterns of neural activity observed during natural sensory processing. Intuitively, a biomimetic code would be able to provide a sufficiently rich and easily interpreted artificial sensory feedback signal. Preliminary studies using biomimetic stimulation show promise, at least for artificial tactile feedback (Berg et al. 2013; Tabot et al. 2013). However, it not clear how well this approach will extend to proprioception, especially in the context of SCI.

First, it is not possible to simply invert the biophysics of neural activity, i.e. to precisely recreate recorded patterns of activity. Each pulse of electrical current activates a sphere of neural activity around the stimulation site (Stoney et al. 1968; Tehovnik 1996; Tehovnik et al. 2006) or, according to another study, a sparse, distributed population of cells whose processes lie proximal to the stimulation site (Histed et al. 2009). In either scenario, it would be difficult to target individual neurons without undesired and/or unpredictable collateral activity (Butovas and Schwarz 2003). This problem will be particularly acute in brain areas that lack a fine-scale topographic map, i.e. where nearby neurons have different response properties. This appears to be the case in the proprioceptive regions of primary somatosensory cortex (S1) (Kaas et al. 1979; Weber et al. 2012). Even if very small currents could be used to activate single neurons (Houweling and Brecht 2008), the total number of (directly) activated neurons would be limited by the size of the stimulating array. In the near term, then, it seems infeasible to create targeted spatiotemporal patterns that mimic the precision and complexity of natural sensory activity.

A second obstacle faced by biomimetic approaches is our piecemeal understanding of how those natural patterns activity encode the sense touch (Johansson and Flanagan 2009), proprioception (Prud’homme and Kalaska 1994), and their complex interaction (Rincon-Gonzalez et al. 2012; Warren and Tillery 2011). Even in cases where elements of this coding are known—e.g. topographic maps for the digits—SCI with sensory loss raises the additional challenge that the code cannot be simply mapped using natural stimuli. While the problem could be inverted—stimulating individual sites and asking for reports of the subsequent percept—this approach would be slow and coarse.

A third challenge for the biomimetic approach is the cortical remapping that occurs after the loss of sensory afferents. Somatosensory perturbations elicit plastic changes in the adult brain in a matter of months, so that cortical representations of deafferented surfaces become occupied by expanded representations of the surrounding areas (Merzenich et al. 1983a, b). For example, following complete loss of sensory input from the hand, the cortical hand representation ultimately represents somatosensory input from the face (Pons et al. 1991). Further complicating the matter, cortical stimulation itself alters the topography of the sensory cortex (Recanzone et al. 1992), shifting the receptive fields of neighboring cells towards that at the site of stimulation. Thus, neural plasticity must be considering in designing systems for artificial somatosensory feedback: in the absence of a stable cortical map, the target for biomimetic stimulation may itself be non-stationary.

Of course, these forms of neural plasticity could also facilitate artificial somatosensation—for example, allowing the brain to reorganize to use new sensory signals when they are provided (Frey et al. 2008). Indeed, we argue next that precise biomimetic stimulation may not be needed to achieve artificial sensory feedback. Rather, we suggest taking advantage of ongoing neural plasticity, which should permit the brain to find useful signals in even non-biomimetic patterns of stimulation.

3 Harnessing Neural Plasticity

Despite the difficulties associated with biomimetic stimulation, there is both aesthetic appeal and, possibly, practical expediency in trying to mimic the way that natural systems work. Here we introduce a complimentary approach in which we focus not on how information is naturally encoded in the brain—or how to precisely reproduce that code—but rather on how the brain naturally learns to use new sensory information and how we can harness that process.

This approach should be made possible by the (perhaps counterintuitive) fact that the brain is always learning to use sensory information. Experimental evidence from our lab (Verstynen and Sabes 2011; Sober and Sabes 2005; Simani et al. 2007; McGuire and Sabes 2009) and from others (Ernst and Banks 2002; van Beers et al. 1999; Gu et al. 2008; Burge et al. 2010) demonstrates that sensory streams are integrated to provide statistically efficient feedback, and this process involves continuous recalibration. As described in more detail next, we propose that this ongoing neural plasticity is the key to developing a system for artificial somatosensory feedback. In particular, we propose that with the right training regime, the brain can learn to interpret and use novel artificial sensory signals, even when those signals are not biomimetic.

As an example, consider the problem of reaching to touch an object. To accomplish the reach, the brain must combine sensory estimates of the hand and the object into the appropriate motor command. Humans (van Beers et al. 1999) and animals (Gu et al. 2008) naturally integrate multiple streams of sensory information (e.g., vision and proprioception) about relevant parameters (e.g., position, velocity, etc. of the hand) into a unified estimate. Experimental evidence indicates that—at least to a first approximation—this is a statistically optimal process, in that the individual modalities are combined in order to minimize the variance of the integrated sensory estimate (van Beers et al. 1999; Ernst and Banks 2002; Gu et al. 2008; Fetsch et al. 2012; Alais and Burr 2004). This can be expressed mathematically as

$$\hat{x}_{int} = \left( {\frac{{\hat{x}_{1} }}{{\sigma_{1}^{2} }} + \frac{{\hat{x}_{2} }}{{\sigma_{2}^{2} }}} \right)\sigma_{int}^{2} ,$$
(1)

where \(\hat{x}_{1}\), \(\hat{x}_{2}\) and \(\hat{x}_{int}\) are the mean estimates from the individual modalities and the integrated estimate, respectively. The trial-to-trial variances of these estimates, \(\sigma_{1}^{2}\), \(\sigma_{2}^{2}\), and \(\sigma_{\text{int}}^{2}\), can also be viewed as measures of the sensory uncertainty associated with a given estimate. Note that the integrated estimate places more weight on the modality that has the smaller variance, or least uncertainty. The variance of the integrated estimate, \(\sigma_{\text{int}}^{2}\) is given by Eq. 2,

$$\sigma_{int}^{2} = \left( {\frac{1}{{\sigma_{1}^{2} }} + \frac{1}{{\sigma_{2}^{2} }}} \right)^{ - 1} ,$$
(2)

is guaranteed to be smaller than the variances of the individual estimates, as long as they are finite. This decrease in variance indicates one of the advantages of multisensory integration, and highlights one reason why artificial somatosensation is expected to improve the performance of neural prostheses.

There is preliminary experimental evidence that, at least for some sensory modalities, the coding of multisensory spatial variables reflects minimum-variance integration (Fetsch et al. 2012; Gu et al. 2008), yet the neural mechanisms underlying integration are net yet known. Similarly, although we know that integration is a learned process (Wallace and Stein 1997) and that it continually recalibrates itself (Simani et al. 2007; Burge et al. 2010), the neural basis for these plastic changes is not known.

Our lab has recently proposed a novel model of adaptive multisensory integration in a neural network (Makin et al. 2013). The model stems from the idea that multisensory integration can be viewed as one example of a more general, unsupervised learning problem, namely latent variable density estimation (LVDE). The goal of LVDE is to extract low dimensional representations of incoming data while retaining as much of the original statistical structure as possible. Our model is implemented with a simple neural network that learns LVDE via a biologically plausible Hebbian-like learning rule (Hinton et al. 2006). This model is illustrated in Fig. 1a. It consists of two populations of input neurons—“visual” neurons encoding hand position in extrinsic coordinates, and “proprioceptive” neurons encoding hand position in terms of joint angles—and an output population of multisensory neurons that receive projections from the two input populations. Starting from a state of random connectivity, the links from input to output are learned through exposure to data in which a strong correlation between the two input populations arises from the fact that they both represent the same underlying variable(s), x, for example the state of limb. After learning, the network is able to perform minimum-variance cue combination, as well as a range of other movement-related multisensory computations (Makin et al. 2013).

Fig. 1
figure 1

Learning to integrate natural and artificial sensory signals. A schematic of a neural network model that learns, de novo and in an unsupervised fashion, to integrate visual and proprioceptive feedback of a variable of interest, x, such as the position or velocity of the limb. Learning is driven by correlations between the input populations, which in turn reflect their common encoding of the variable x. The same system would be equally able to learn novel artificial sensory inputs, assuming the inputs remain correlated. This figure is adapted from Makin et al. (2013)

Full size image

A key insight from the model in Fig. 1 is that the statistical properties of the input signals, namely the correlation between the activities of the two sensory neural populations, are sufficient to drive the network to learn integrated representations of hand position (Makin et al. 2013), without the need for supervisory signals. We do not know if this model accurately captures the mechanisms implemented in the brain, but its biologically plausible form makes it an exciting candidate and, moreover, it makes several testable predictions of practical importance for BCI.

A testable prediction that is particularly relevant for artificial somatosensation is that correlation between two input signals will drive learning and integration. This suggests a powerful learning-based approach for delivering novel sensory signals to the brain: if one provides a novel stream of information, delivered for example by intracortical mircrostimulation, the brain should learn to interpret and integrate that signal as long as it correlates over time with a known sensory signal. Specifically, we propose to deliver an informative artificial feedback signal to a somatosensory area and to correlate this signal with vision. This pairing will drive learning and, ultimately, integration of the two signals, mimicking natural sensory processing.

4 Candidate Neural Structures to Target

In the preceding section we argued for a new, learning-based approach to providing artificial somatosensory feedback, in which cortical stimulation is paired with visual feedback. Here we address the question of where in the brain the stimulation should be delivered. To inform this discussion, the principal cortical areas involved in goal-direction reaching are illustrated schematically in Fig. 2. Briefly, sensory information ascends from the periphery via the thalamus to modality-specific primary sensory areas. These, in turn, project to a number of multisensory cortical areas in the parietal lobe. Motor planning appears to occur across this parietal circuit and in the interconnected pre-motor and primary motor cortex in the frontal lobe. The ideal target for somatosensory stimulation would seem to be one that either lies upstream of, or is one of, the multisensory areas involved in movement planning (Fig. 2).

Fig. 2
figure 2

Possible cortical targets for artificial somatosensory signals. Schematic of the cortical circuit underlying sensory integration and reaching. Visual and proprioceptive information enter the parietal cortex via different pathways, and they are integrated before the information is passed to pre-central motor cortex

Full size image

Neurons in primary somatosensory cortex (S1) process both touch and proprioceptive information. In primates, S1 is composed of cortical areas 1, 2, 3a, and 3b. Area 3a is largely proprioceptive; 3b mostly cutaneous; and areas 1 and 2 have mixed responses (Krubitzer and Kaas 1990). All of these subdivisions are good candidates for providing artificial somatic sensations, but areas 1 and 2 offer an important practical advantage: easier targeting with electrode arrays because of their superficial location on bank of the post-central gyrus. In contrast, areas 3a and 3b are less accessible: 3b lies on the bank of the central sulcus and 3b is located near its fundus.

Importantly, electrical stimulation of S1 produces sensations referred to the body (Penfield and Boldrey 1937). Stimulating in S1 (and the thalamic regions that project to it) produce sensations at lower thresholds than “higher” cortical areas (Doty 1969). Additionally, S1 projects directly to areas that are known to perform multisensory integration (A5, PMd) as well as to primary motor cortex (Fig. 2). The latter connection, in particular, may play an important role in the ability of somatosensory feedback to elicit rapid corrective responses during movement execution (Omrani et al. 2013). For these reasons, S1 is likely to play a privileged role in providing sensory feedback for BCI control, both for integrating with vision and for establishing a short latency loop between motor output and sensory feedback. Furthermore, neurons in S1 are known to be plastic; for example, they can develop visual responses through somatosensory-visual pairing (Shokur et al. 2013). For these and other reasons, most cortical approaches to providing artificial somatic sensation have targeted S1 (London et al. 2008; O’Doherty et al. 2009, 2011, 2012; Berg et al. 2013; Tabot et al. 2013).

The parietal lobe also contains many multisensory areas. Of these, area 5, which receives projections from S1, may be particularly well suited, as it is situated conveniently on the cortical surface and seems involved in encoding arm postures and movements relative to the body in the context of upcoming motor responses (Lacquaniti et al. 1995; Kalaska 1996). Furthermore, in intact animals, area 5 may encode feedback from online perturbations during movement execution even earlier than S1 (Omrani et al. 2013). Neurons in area 5 also have larger and more complex receptive fields than S1 (Hyvarinen 1982), suggesting that it employs a more complex, or abstract, representation of proprioceptive space. Finally, area 5 projects directly to dorsal premotor cortex (PMd), and, along with S1, to primary motor (M1) cortex.

Lastly, we note that the thalamus—an earlier subcortical stage of sensory processing—is also an attractive target for stimulation. The percepts evoked by thalamic stimulation correspond to relatively small and localized portions of the body surface (Heming et al. 2011), implying that each channel on a stimulating array could be used as a distinct input channel. Additionally, the neural activity evoked by in cortex by stimulation in the thalamus may more closely resemble “natural” neural activity than that evoked by stimulation of cortex directly (Brockmeier et al. 2012; Choi et al. 2012). Indeed, taking advantage of the information processing performed by the thalamocortical projections may represent the key to obtaining more biomimetic activity patterns in S1. Unfortunately, however, the thalamus is difficult to reach using currently available electrode arrays, and thus far, it seems that evoked percepts more often resemble a numbness or tingle than touch or movement (Heming et al. 2010).

5 Conclusion

In the preceding, we have argued that artificial somatosensory feedback will be needed to optimize clinically viable neural prostheses. Furthermore, we argued that it is practical to provide this feedback through the electrical stimulation of the brain, and indeed that this may be the only viable route for some patient populations, including following SCI with sensory loss. Lastly, we suggested a novel, learning-based approach to providing artificial somatosensory feedback.

Importantly, we are not arguing that the learning-based approach excludes more traditional biomimetic paradigms for stimulation-based feedback. Indeed, we think it is likely that the learning-based system would be learned more quickly as the stimulation protocol drives increasingly biomimetic patterns of activity. Rather, we are arguing that success of this approach need not rely of achieving high-fidelity biomimetic stimulation: the brain’s natural mechanisms of plasticity should be able to make up for the difference.