Introduction

The number of blind individuals has been increasing around the world and the USA. Globally, 32.4 million people were blind in 2010 which has increased by 600,000 since 1990 [1]. In 2015, a total of 1.02 million people were blind in the USA and it is projected to double to approximately 2.01 million people by 2050 [2]. The negative impacts of this condition on individual physical and mental health include a higher risk of chronic health conditions [3], unintentional injuries [4], social withdrawal [5], depression [5, 6], and mortality [7, 8]. From a socioeconomic standpoint, blindness has a negative effect on educational and occupational opportunities as well as medical expenditure. In the USA, the excess medical expenditure (compared to individuals without visual impairments) was $2.5 billion in 2004 which was primarily attributed to homecare [9]. Therefore, any effort leading to improvement in day to day life of these individuals not only improves their quality of life but also relieves a significant economic cost.

Common causes of blindness include cataracts, glaucoma, macular degeneration, infections, vitamin A deficiency, diabetes, and trauma [10]. Although in some of these conditions, blindness can be prevented in earlier stages [11,12,13,14]; many cases lead to irreversible loss of vision. Sensory substitution is an extensively explored approach to channel some degree of information via non-visual senses such as tactile [15, 16] or audible signals [17, 18]. Although sensory substitution is non-invasive and easily available for majority of blind individuals, the quantity and quality of information perception are extremely limited [19]. Hence, a more direct approach may be helpful to generate a visual perception of the surrounding environment.

Visual neural prostheses represent an approach of generating visual perception via direct stimulation of visual pathway. When the light is projected on the retina, optical information is converted to neural spiking activity via photoreceptor cells, transmitted via retinal ganglion cells to the lateral geniculate nucleus of the thalamus, and conveyed to the primary visual cortex via thalamic synapses. The basic principle of visual prostheses is that any segment of the visual pathway can potentially interface with a prosthetic that captures the image of the visual field via a camera, properly maps it into temporally and spatially specific signals, and stimulates the visual pathway accordingly. Stimulation in its simplest form induces perception of a spot of light called a phosphene and it could be used as a building block to construct a representation of the visual scene which conveys helpful information for communication and mobility. The site of stimulation is dependent of the pathology of the blindness and is chosen in a way that bypasses the damages segment of the visual pathway. Current visual prosthesis approaches include stimulation of the retina, optic nerve, lateral geniculate nucleus of the thalamus, optic radiations, and visual cortices. The number of subjects eligible for different types of prostheses remains unclear and requires further characterization, but considering retinal and subretinal prostheses deliver sensory input at earlier parts of the visual pathways than cortical stimulation, it is likely safe to assume that more patients stand to benefit from developing strategies that completely bypass the early visual pathway.

Our focus in this review is to elucidate the current state of cortical prostheses for visual restoration. However, it is important to acknowledge other strategies that target earlier segments of the visual pathway. Retinal prostheses aim to stimulate the surviving population of inner retina, retinal ganglion cells, and/or bipolar cells in individuals with retinis pigmentosa or age-related macular degeneration [20]. In theory, such an approach is the most advantageous as stimulation occurs at the earliest stage of visual perception, prior to any integration or “post-processing” of visual signals in the brain, especially the lateral geniculate nucleus. Argus II (Second Sight Medical Products) is the first commercially approved retinal prosthesis using the epiretinal approach, in which the microelectrode array is implanted on the retinal surface near the nerve fiber directly stimulating the ganglion cells [21]. Argus II improved subjects performance in spatial-motor tasks [22, 23], detection of motion [24], reading [25, 26], and face recognition [27]. Subretinal implants are another approach for retinal prosthesis in which the electrode array sits on the outer surface of the retina relying on normal processing of inner and middle retinal layers [28]; Alpha IMS is a subretinal implant that has been approved for commercial use in Europe [29] and is shown to restore some level of light perception, motion detection, visual acuity, and reading ability [30,31,32]. Retinal implants are currently the only visual prostheses that have approval for commercial use; however, potential candidates for such devices are a small portion of the blind population [33,34,35,36,37,38].

Stimulation of the optic nerve has also been shown to elicit phosphenes [39]. Veraart et al. utilized a spiral cuff electrode with four contacts. Subjects were able to discriminate the orientation of lines without error and recognize simple patterns with a recognition score of 63% [40]. This approach however requires an intact optic nerve and may not be applied to a large population of blind individuals. Deep brain stimulation of thalamus in proximity of lateral geniculate nucleus (LGN) and optic radiations has been resulted in reports of visual sensations [41]. Microstimulation of LGN has been investigated in animal models based on visual saccades [42, 43]. However, phosphene perception in human trials is yet to be explored.

Stimulation of early areas of the visual cortex (V1, V2, and V3) elicits perception of phosphenes and is used to develop cortical visual prosthetics. Our main objective in this review is to extensively address this cortical stimulation approach and provide an insight on the current state as well as current challenges of visual cortical prostheses.

History

Eliciting visual perception via electrical stimulation of the visual cortices was first reported in a case study by Löwenstein and Borchardt in 1918 where wounded soldiers were able to see flickering light perceptions on the opposite half of their visual field after stimulation of occipital lobe [44]. This was then followed by the work of Förster and Krause who were able to expose the human occipital lobe under local anesthesia and reliably induce perception of phosphenes [45, 46]. These studies set the groundwork for the first systematic effort to develop a visual cortical prosthetic by Brindley and Lewin [47]. They implanted a blind subject with an intracranial array of 80 electrodes molded it to fit the calcarine sulcus and neighboring cortex of the right hemisphere. They were able to successfully elicit phosphenes in different areas of the visual field which provided a proof of feasibility for visual cortical prosthetics. This array was physically connected to an extracranial array of radio receivers that allowed for selective activation of intracranial electrodes.

Dobelle reported similar observations on a group of 15 sighted patients who were going under surgery for other clinical reasons and gathered a useful body of data on the proper electrode placement as well as the effects of electrode size and stimulation parameters on the quality of perceptions [48]. Subsequently, they implanted two blind individuals with a 64-electrode array and were able to elicit simultaneous phosphenes representing simple patterns including a square and a reverse letter “L.” [49] In another experiment, they were able create English letters using six electrodes with phosphenes scattered in the visual field [50]. Using a combination of sequential and simultaneous stimulation of these six electrodes, they developed a “cortical braille” that was able to present randomized alphabets and synthetize sentences. One of the subjects performed with up to 85% accuracy on recognizing the alphabets and was able to read short sentences with missing or misreading a few words. Moreover, they developed a portable version of the device that was capable of receiving input from a video camera [51]. They reported that the subjects were able to read a 6 in. square “tumbling E” at 5 ft, as well as Snellen letters, HOTV test, Landolt rings, and Lea figures of similar size.

While Dobelle was making progress on improving the functionality of prosthetics with subdural electrode arrays, the NIH formed the neural prosthesis program that took another approach to decrease the current threshold required for eliciting phosphenes [52]. Based on a previous study on non-human primates [53], they proposed that intracortical stimulation of striate cortex can generate more stable phosphenes with lower current thresholds. They inserted penetrating electrodes into the occipital lobe of three sighted volunteers whom were undergoing occipital craniotomies under local anesthesia. All subjects reported perception of phosphenes elicited by currents (10–300 μA) significantly lower of the threshold amount for subdural electrodes (600–2000 μA). Following their success in sighted individuals, they implanted penetrating microelectrodes into the occipital lobe of a blind individual [54]. The microelectrode leads were assembled into groups of 8 or 10 and connected to the cables that exited the scalp through four separate incisions. Thirty-four out of 38 electrodes were able to elicit phosphenes with threshold currents of 2–77 μA at 200 pulses per second (twice the rate of stimulation used by Bak et al.). They characterized the effects of stimulation parameters on current thresholds and showed that an increase in pulse width and duration of stimulation decreased the current thresholds. Unlike most cases reported by Brindley and Lewin, the phosphenes were not perceived as flickering; they determined the interval between the two consecutive pulses needs to be shorter than 25 ms in order to generate a persistent (non-flickering) perception. Increasing current amplitudes had different effects depending on the site of stimulation, resulting in larger phosphenes in some electrodes, smaller in others, and no change in others. They also reported that phosphenes could be perceived in different colors (blue, red, violet, yellow but never green), with suprathreshold stimulation tending to elicit white or yellow phosphenes. They explanted the device after 4 months and concluded that their experiments demonstrate the feasibility of intracortical visual prosthetics.

Current State

Today, several teams are working towards understanding perceptual effects of visual cortex stimulation and development of safe and reliable visual cortical prostheses. Epilepsy patients with electrodes implanted in the occipital lobe provide a valuable opportunity to study the stimulation effects of visual cortex [55]. Functional mapping of the visual cortex [56,57,58,59,60] and understanding the psychophysical correlates of phosphene perception [61, 62] in these patients have extended our understanding of artificial vision and is providing an extremely valuable insight for development of visual cortical prostheses.

Multiple sites across the world are currently developing and testing visual cortical prosthetics with a general design similar to the most recent design of Dobelle [51]. As depicted in Fig. 1, the visual information is captured by a camera located on a glass or head band [63]. Pixel information is then fed to a video processing unit (VPU) and converted to a continuous sequence of commands that activate a subset of electrodes and determine the stimulation parameters (amplitude and waveform) [63]. The commands are transferred to an implanted system that delivers patterned stimulation. While Dobelle and others achieved promising outcomes with their systems, one of the main disadvantages of their design was the risk of infection and device breakage due to chronically externalized connections [64]. Hence, current designs take advantage of the state of the art Application Specific Integrated Circuits (AISC) and coil fabrication techniques to create a wireless data and power link between internal and external units [65]. Similar to previous works, both intracortical and surface stimulation approach are being investigated. Here, we provide a brief description of multiple projects that are currently in the device development or human experimentation phase.

Fig. 1
figure 1

Components of a cortical visual prosthesis (with permission from SecondSight Inc.)

Full size image

The ICVP Project

The NIH neural prosthesis program ended in 1996 and reconstituted in 2000 as the Intracortical Visual Prosthesis Project (ICVP) at the Illinois University of Technology. While maintaining the intracortical stimulation approach, they developed a Wireless Floating Microelectrode Array (WFMA) to eliminate the use of wires and chronic incisions [64]. WFMA consists of 16 microelectrodes with different lengths located on a 2 mm by 2 mm ceramic substrate platform (Fig. 2) [63]. An application specific integrated circuit (AISC) along with a micro-coil is embedded in the platform to facilitate wireless power and datalink between the electrodes and an extra-corporal telemetry controller (TC). The TC is able to communicate with multiple implanted WFMAs and control each electrode individually.

Fig. 2
figure 2

WFMA structure including an ASIC (chip), microelectrodes (shorter pins), and wireless coil located on top of the ASIC providing transcutaneous inductive coupling (with permission from Laboratory of neural prosthetic research at IIT)

Full size image

Clinical testing of ICVP project has been planned but is not started yet [63]. The available anatomical areas are the occipito-dorso-lateral surface and posterior inferior gyrus while placement of electrodes on the wall of medial occipital lobe will be avoided to minimize the surgical risks. This limits the exposed area for V1 and increases the chance of electrode placement over V2 and V3. For the early recipients of the device, the implantation of the WMFAs would be restricted to one hemisphere and about nine modules (144 electrodes). The anticipated components of experimental trials are phosphene threshold testing and intensity rating, phosphene mapping, and testing the ability to convey basic visual percepts such as simple geometric.

CORTIVIS

Cortical visual neuroprosthesis for the blind (CORTIVIS) is a multi-site project supported by the Commission of the European Communities aiming at developing an intracortical visual cortical prosthesis [66]. CORTIVIS takes advantage of the Utah electrode array (UEA) which has been FDA approved for long-term human studies. UEA consists of 100 narrow silicon microelectrodes protruding 1.0–1.5 mm from a flat rectangular 4 mm by 4 mm base that lays on the surface of the lateral occipital cortex (Fig. 3) [67]. Similar to the ICVP project, an implanted AISC receives the stimulation parameters and power via wireless link from an external RF block connected to a pocket-size processor [68].

Fig. 3
figure 3

Overview of the implantation site of the electrode array on the surface of the lateral occipital cortex [66] (with permission from the author)

Full size image

Although clinical trials for the CORTIVIS project have not started, preliminary investigations have been conducted on epilepsy patients to establish the safety of the implantation procedure as well as phosphene assessment [69]. All subjects perceived phosphenes and tolerated the procedure without complications. Stimulation of both early visual areas and extrastriate occipital cortex induced phosphenes with retinotopic representation. Phosphenes were described as flashing, colored or uncolored lights and stars and their sizes ranged from a “pinpoint” to occupying almost the whole visual field depending on the location and stimulation parameters.

Gennaris

The Gennaris bionic vision system is an intracortical visual prosthesis developed by Monash Vision Group supported by the Australian Research Council. Similar to the ICVP project, Gennaris uses a number of autonomous implant tiles to cover the visual cortex [70]. Each ceramic tile holds 43 electrodes protruding from its base and contains a receiver coil and an ASIC. A custom-designed “Pocket Processor” extracts the useful features from the image captured by the camera and generates a pattern of electrode activation to create the most helpful information for the user depending on the task at hand [71]. They introduced the concept of transformative reality (TR) that instead of being restricted to the direct visual representation of a scene asks the following question: “What combination of sensors, processing and rendering gives the most usability?” [72]. This could be as a simple as an edge detection algorithm to present the boundaries of the objects or it can integrate additional data from the environment into the image and use more complicated image processing and machine learning algorithms. For example to help patients with navigating cluttered indoor areas, they developed a TR mode that renders the patches of empty ground as phosphenes. The algorithm detects the ground plane using RANSAC plane fitting and real-time accelerometer data by estimating the direction of gravity.

Although the clinical trials for Gennaris have not started yet, safety and efficacy of the system have been verified in multiple ways. Chronic implantations and stimulation of electrodes were performed in rat motor cortex and electrode impedances were measured along with the stimulation thresholds required to evoke motor output [73]. Electrode impedances increased over the first 4 weeks of implantation and stabilized at weeks 5 and 6. Histology of the brain tissue collected after the implantation period demonstrated that healthy neurons were in close proximity to the neurons and there was minimal astrogliosis. Moreover, a sheep model was used to demonstrate the surgical implantation of Gennaris tiles. Preliminary data indicated the minimal damage to the cortex and wireless transmission of power and data between the external coil and implanted tiles were achieved [70]. Similar to the other intracortical electrode arrays, Gennaris tiles will neither be placed on the medial aspect of the hemispheres nor within the calcarine sulcus to prevent the damage to the adjacent cortex and ensure proper wireless communications. The initial plan is to insert up to six tiles (258 electrodes) above and below the calcarine sulcus. Psychophysics and patients training phase of the study will start with phosphene mapping and will followed by the integration of camera input.

Orion

Unlike other devices, the Orion device (Second Sight Medical Products) uses a surface grid for subdural stimulation of the visual cortex. Argus II is a retinal prosthetic device developed by the SecondSight that is commercialized and widely used specially in the USA. Orion inherits many features of Argus II including the external components (camera, VPU, and transmitter coil) and frameware. The implantable components of Orion include a receiver coil and internal circuit that sits on the skull (Fig. 4) and controls the stimulation parameters delivered by the electrodes. The subdural electrode grid consists of 60 surface electrodes placed on the medial occipital lobe spanning a portion of calcarine sulcus.

Fig. 4
figure 4

Electrode placement and implantation site for Orion (with permission from SecondSight Inc.)

Full size image

Orion is the only visual cortical prosthesis that has started the clinical trials. A preliminary study was performed by our group on a blind individual using an off-the-shelf neurostimulator at the University of California Los Angeles. Two 4-contacts subdural electrode strips were implanted over the right medial occipital lobe spanning a portion of the calcarine sulcus. Electrode strips were connected to a Neuropace RNS stimulator (NeuroPace RNS Inc., Mountain View, CA) placed within the crainium coplanar with the skull surface and was covered by the scalp. Phosphene mapping and threshold testing were performed for a period of 18 months and there were no clinical complications during and post implantation. Phosphenes were elicited by stimulation via all contacts and were scattered over the left side of the visual field. After confirming the safety and feasibility of cortical surface stimulation in generating phosphene perception, Orion received FDA approval for clinical trials. The device was implanted in five blind individuals at University of California Los Angeles and Baylor College of Medicine and all subjects reported perception of phosphenes. Psychophysical tests including phosphene mapping and threshold testing are currently being conducted and initial integration of camera with the stimulation is being tested.

Challenges and Potential Solutions

Subject Selection

Although progress in development of visual cortical prosthesis is accelerating in the recent years, more work needs to be done to achieve a practical device. Hence, it is not ethical to expose patients with any potentially functional residual visual to invasive brain surgery as the potential risk of loss of residual vision can have severe functional impacts. Therefore, performing visual function tests is an essential component for recruiting potential subjects as they estimate the subject’s capability of performing real life tasks using his/her residual vision if any. Moreover, these tests can provide a quantified measurement of visual function which can be acquired at various time points during the clinical trials and determine the device performance. For example, square localization, direction of motion, and grading visual acuity are visual function tests that have been sued for potential recipients of retinal prosthetics as well as clinical trials for cortical prostheses [74]. Also subjects who are qualified for retinal implants have a significantly higher chance of benefiting from retinal prostheses and are not proper candidates for cortical implants [74,75,76].

Electrode Array Design

One major dilemma in designing cortical prosthetics is the choice between intracortical or surface electrodes. While both techniques have previously proven to be feasible, each has its own limitations.

Spatial Specificity

Surface electrodes have relatively larger surface area and therefore require higher electrical currents to elicit phosphenes [52]. In theory, when multiple electrodes are stimulated, these large currents can interact in a nonlinear fashion evoking phosphenes with spatial properties different from what each electrode evokes individually [66]. However, Bosking et al. have shown that concurrent surface electrode stimulation of two sites separated by greater than 6–10 mm reliably elicits two phosphenes that are spatially close to where stimulation of each single electrode would elicit [61]. Also configuration of three electrodes appeared to largely follow the same rules and they could stimulate up to 4–6 electrodes and achieve perception of up to five phosphenes. Although these results do not contradict with the problem of spatial distortion, their findings indicate that with certain considerations (such as the distance between stimulated electrodes) evoking concurrent phosphenes with predictable spatial pattern is possible using surface electrodes.

Long-Term Stability

While penetrating microelectrodes provide higher spatial resolution with lower current thresholds, highly invasive nature of this approach undermines their long-term reliability. The electrical, mechanical, and biochemical mismatch between the electrodes and cortical tissue causes unwanted tissue responses including neuronal loss and glial encapsulation which can compromise stimulation capability of the device [77,78,79,80,81,82]. Subdural electrodes on the other hand do not require penetration of the cortex and have greater long-term stability [83, 84]. This allows them to be a reliable candidate for many long-term applications including BCIs [85, 86] and treatment of epilepsy [87, 88].

V1 Accessibility

Retinotopic maps of visual cortex reveal that a large portion of early visual areas (V1, V2, and V3) is located in the medial surface of occipital cortex [89]. So far, none of the microelectrode arrays have been or are planning to be implanted in the medial wall of the occipital lobe for two reasons: a) Risk of damage to the adjacent cortex and b) difficulty in wireless transmission when the receiver and transmitter coils are orthogonal [63, 66, 70]. Therefore, the occipito-dorso-lateral cortex is the preferred implantation site for microelectrode arrays and it might cover higher level visual areas (hMT+/V5 and LOC) where the likelihood of phosphene induction is relatively lower [90]. Moreover, this can potentially limits the eccentricity of the phosphenes as the ventro-medial areas of the occipital cortex is associated with the periphery of the visual field [91, 92].

Psychophysical Testing

Psychophysical testing provides a benchmark for a preliminary evaluation of the device performance and allows for gathering essential data for creating complex percepts [93]. Obtaining threshold levels and phosphene locations for each electrode are the two major components of psychophysical testing and are required for proper integration of camera with the implant.

Threshold Testing

Phosphene threshold is the minimum amount of current or charge required to elicit a phosphene. Thresholds may vary across the electrodes and depend on the stimulation parameters (frequency, pulse width, and burst duration). For instance, Dobelle et al. reported current thresholds for surface electrodes in the range of 1–4 mA for pulsewidth of 0.25 ms and frequency of 50 Hz [92]. He also showed that increasing pulsewidth and frequency tends to decrease the threshold levels while electrode size did not have a significant correlation with current thresholds [48]. Determining thresholds for each electrode allows of optimization of electrical stimulation delivery by minimizing the energy consumption and damage to the underlying tissue [94, 95]. Moreover, small phosphenes are often more desirable as they allow for creating patterns with higher spatial resolution. Phosphene size has been shown to increase for higher current levels [62]; therefore, stimulation at the threshold levels can potentially generate smallest phosphenes for each specific electrode.

Phosphene Mapping

The spatial representation of visual field is preserved and repeated multiple time across visual cortices, especially in the early visual areas [89]. Subject-specific retinotopic mapping of early visual areas can be obtained using functional MRI and projected on the unfolded cortical surface [91]. However, this requires some degree of visual function hence it is not applicable to blind individuals. Therefore, interactive phosphene mapping after implantation of visual cortical prosthesis is a necessary step to generate spatially relevant visual sensation. The most basic paradigm of phosphene mapping involves stimulation of each electrode and recording the location of the phosphene based on subjects feedback.

One of the challenges in collecting phosphene mapping data is the phosphene displacement in response to eye movement [39, 49, 54]. When a phosphene appears in the periphery of the visual field, subjects tend to track the phosphenes with their eyes. This results in the perceived phosphene location following the direction of eye movement using optomotor feedback [96]. To compensate for this effect, subjects are asked to focus on the center of the FoV using a tactile or an audio cue and maintain their gaze during stimulation. This consideration however may need to be maintained during daily usage of the device and subject needs to be trained accordingly. This issue can potentially be addressed by eye tracking and adjusting the phosphene mapping accordingly. This solution may not however be applicable to patients with enucleation or other traumatic causes of blindness which result in an absent globe.

Machine Learning

Measuring threshold levels and phosphene mapping provide essential data for the integration of camera and the neurostimulator to form a brain computer interface (BCI). A visual prosthetic BCI in its simplest form maps the light intensities in the visual field to charge/amplitude levels deposited on the surface of the primary visual cortex. This however may be problematic given the limited number of electrode and nonuniform arrangement of their corresponding phosphenes in the visual field. Incorporation of machine learning into the BCI could potentially improve the device performance in different ways. The TR approach being developed by the Monash group is an example of using machine learning for image processing to produce visual perception optimized for the specific task in hand such as navigation or face detection.

There have been significant advances in decoding intracranial recording of the visual pathway to detect and discriminate the complex visual perceptions using machine learning [97,98,99]. Moreover, stimulation of cortical areas further down the visual processing stream affects higher level and complex visual perceptions [100,101,102]. For instance, stimulation of fusiform gyrus is reported to alter face perception [103, 104] and reading capabilities [105, 106]. Although our understanding of this approach is too limited to produce relevant visual information, given the to-down and recurrent organization of the visual system [107, 108], it could potentially serve as a supplement to earlier visual cortex stimulation to produce a rich and more natural visual experience.

Conclusion

Development of cortical visual prosthetics has gained a great momentum in the recent years and based on current advances in electrode design and wireless communication, it is expected to restore some degree of vision in blind individuals. In this paper, we briefly reviewed the history and described the state of the art theses prostheses aiming to provide a basic understanding of the approach and potential challenges.