Raising the stakes for cortical visual prostheses

In this issue of the JCI, the dream of restoring useful vision to blind individuals with neurotechnology moves one step closer to realization. Fernández et al. implanted an electrode array with 96 penetrating electrodes in the visual cortex of a blind patient who had been without light perception for 16 years due to optic neuropathy. Remarkably, the patient was able to perceive visual patterns created by passing current through array electrodes. The use of a penetrating electrode array meant that action potentials from single neurons could be recorded to study the neural response to stimulation. Compared with electrodes resting on the cortical surface, penetrating electrodes require one-tenth the current to create a visual percept. However, patterned electrical stimulation often fails to produce the expected percept for penetrating and surface electrode arrays, highlighting the need for further research to untangle the relationship between stimulus and perception.

Neurosurgeons probing the brains of awake patients in the operating room discovered that electrical stimulation of early visual cortex produces the perception of a flash of light, known as a phosphene (1). This discovery prompted attempts to develop a cortical visual prosthesis (CVP) in the 1960s and 1970s, based on the principle of stimulating sets of electrodes to produce combinations of phosphenes that serve as rudimentary visual images for blind patients (2–4). However, development stalled because of the primitive state of neurotechnology; huge racks of equipment were required for stimulation and hardware failures were common. Advances in neurotechnology have recently spurred a renaissance of interest in CVPs (5–7).

There are two categories of electrodes used for CVP design: the array of electrodes may either be placed on top of the cortical surface, or the array may penetrate the cortex. In this issue of the JCI, Fernández et al. (8) used the penetrating approach, choosing a location in early visual cortex in the occipital lobe as the site for implanting a silicon-based 96-electrode grid, a technology referred to as a “Utah array” after the University where it was created (Figure 1 and ref. 9). The Utah array is a logical choice for this application because it contains more electrodes than previous human CVPs and it has successfully been implanted in long-term human studies of brain-computer interfaces (BCIs) for somatosensation and motor control (10–12).

Figure 1

Stimulation of the Utah array in the visual cortex of a blind subject resulted in visual percepts. (A) A Utah electrode array was implanted near the occipital pole in early visual cortex. Stimulation of a single electrode in the array (blue square) produces the percept of a single bright phosphene in the visual field (blue circle). Percept data are based on Fernandez et al. (8). (B) Percepts produced by stimulating single electrodes lie within the lower left visual field. However, the fine structure of the phosphene locations was disorganized. Stimulation of single electrodes along a column of the electrode array led to a disorderly progression of phosphenes in visual space (colored circles with connecting lines; color indicates correspondence between stimulated electrode and phosphene location, not phosphene colors). (C) Simultaneous stimulation of multiple electrodes produced varying results. Some stimulation patterns produced recognizable letters (top) or coherent lines in visual space (bottom). (D) Some stimulation patterns containing neighboring electrodes produced multiple discrete phosphenes. (E) Some stimulation patterns containing noncontiguous groups of electrodes resulted in single percepts of a horizontal line (top, blue electrodes), a vertical line (top, red electrodes), or a letter (bottom). (F) Some stimulation patterns failed to produce recognizable percepts.

For motor BCIs, the electrode array records the intended action of a subject, as encoded in the activity of cortical neurons, and this information is used to control a robotic manipulator. For CVPs, the information flow is reversed; visual information from a camera is processed by a computer and the array is electrically stimulated, driving neuronal activity that results in a conscious percept for the subject. The major contributions of Fernández et al. lie in the middle step of driving neuronal activity in visual cortex using electrical stimulation (8). Use of concurrent stimulation via multiple electrodes to deliver information to the visual cortex is a process fraught with difficulty. One advantage of penetrating electrodes is that they require much less current to drive activity than do electrodes placed on the cortical surface. Careful threshold measurements by Fernández et al. showed that the average current required to create phosphenes was approximately 70 μA (8), less than a tenth of the milliamp-level currents required for surface electrodes (13–15). While one milliamp of current delivered to a single electrode is of little safety concern, simultaneously driving dozens of surface electrodes at this level could lead to tissue injury and unwanted changes in cortical activity. The process also places great demands on the device’s power supply.

Electrode spacing is an important consideration for an effective CVP. The electrodes in the Utah array are on a 400-μm pitch, compared with the 3-mm spacing between the surface electrodes of the Second Sight Orion CVP. In the Fernández et al. study (8), some adjacent electrode pairs produced two distinct phosphenes when stimulated simultaneously, suggesting that dense packing of electrodes could provide a high-resolution CVP.

Fernández et al. used the recording capability of the Utah array to assess the cortical activity evoked by electrical stimulation (8). In a typical recording session, about one-third of the 96 electrodes captured neuronal activity. However, pairing stimulation with recording revealed extensive variability. For some electrodes, electrical stimulation appeared to evoke an increase in local spiking activity of some cells, while in other cases stimulation appeared to evoke a decrease in local spiking. It is not clear if a broader assessment of local population activity, for example by measurement of local field potentials or summation of activity across a number of electrodes, would demonstrate the same variability. Even for a single electrode, a sequence of stimulation trials sometimes elicited differing effects, with early trials evoking a small increase in neural activity (and no conscious percept for the subject) and later trials evoking a larger increase in activity together with a conscious percept. This variability suggests that CVPs may need to adopt a closed-loop stimulation strategy, in which recorded neural activity is used to adjust stimulation parameters on the fly to create the desired neural and perceptual effects.

The biggest mystery raised by Fernández et al. (8) concerns the uncertain relationship between electrical stimulation and the conscious percept experienced by the blind subject. A fundamental assumption of CVPs is that in sighted adults or blind patients who lost vision as adults (like the patient of Fernández et al.) the visual cortex contains a precisely organized, retinotopic map of the visual field, with each adjacent location in the cortex representing adjacent locations in the real world. Stimulating adjacent electrodes should result in the patient perceiving phosphenes that are close together, while stimulating distant electrodes should result in distant phosphenes; simultaneous stimulation of multiple electrodes should produce a pattern of phosphenes corresponding to the shape of the stimulation pattern.

At least some parts of this framework for the development of CVPs may be due for a reexamination. While all phosphenes in the blind patient were clustered near the center of the visual field, as expected from the location of the Utah array, individual stimulation of electrodes along one column of the array did not produce the expected orderly line of phosphenes in visual space (Figure 1B). This haphazard organization of phosphene locations cannot be explained by simple current spread or well-known principles of visual cortical organization, such as the reversal of the visual field sign at the boundary between visual areas, or changes in the cortical magnification factor with eccentricity.

There are several categories of explanations for the disorganized fine structure of phosphene locations. One category is the imprecision of electrical stimulation, which may activate fibers of passage (shifting neural activity from the site of stimulation to a distant location) or evoke activity in competing excitatory and inhibitory neuronal populations. Another source of variability is the ability of a blind patient to accurately report phosphene location. Sighted humans use precise eye movements and eye-hand coordination to localize targets within a fraction of a degree. Eye movements also affect the perceived location of phosphenes in blind subjects, but since they have no external reference frame to use in holding their eyes steady, it is important to measure and account for eye position at the instant that stimulation is delivered (16). Maps of phosphene locations that are less variable (and more in accord with the known retinotopy of visual cortex) can be constructed by requiring blind subjects to place both hands in contact with a tactile fixation point at the start of each trial and to report the location of phosphenes relative to each other rather than in absolute visual space (17). A final possibility is that the map of visual space in the visual cortex of blind subjects contains more local scatter or disorder than in sighted individuals, although this should not be inferred until other sources of error are controlled.

As shown in Figure 1, there was also high variability in the perceptual results from simultaneous stimulation of multiple electrodes; this variability cannot be easily explained by eye movements or reporting imprecision. In the best case (Figure 1C), the percept matched the pattern of stimulation; stimulating a circle of electrodes resulted in the percept of a circle, while stimulating a line of electrodes produced a line. More often, the percept was unpredictable. Stimulating a few nearby electrodes sometimes produced compact phosphenes (Figure 1D), but stimulating similar numbers of widely separated electrodes sometimes produced extended percepts rather than the prediction of widely separated phosphenes (Figure 1E). In some cases, no coherent percept was reported for stimulation patterns that were expected to produce a letter-like percept (Figure 1F). One concern is that these highly variable results arise from the tightly packed electrodes (400 μm) on the Utah array, leading to crosstalk or interference between stimulated neural elements. However, reliable production of visual forms has also been difficult to obtain with concurrent stimulation of surface electrodes with coarser, millimeter-scale spacing (18). Instead, an alternative stimulation strategy involving dynamic, sequential stimulation of electrodes in the desired pattern was necessary for the reliable production of complex percepts (18).

An important area for future investigation is improvement in CVP function over time due to perceptual learning. In the Fernández et al. study, the patient reported recognizing some stimulation patterns on the very first trial, with no learning required, while other patterns did not produce a percept (ref. 8 and Figure 1). It is possible that over thousands of training trials, participants could learn to distinguish a greater variety of patterns, including some that did not produce a percept in initial testing. While the study of Fernández et al. lasted only 6 months (8), too short for a comprehensive study of learning, the Utah array has been implanted for many years in the BrainGate trials (10), demonstrating that there is no absolute safety barrier to long-term studies. The potential contribution of learning to the effectiveness of visual prostheses depends on a host of complex factors, including the degree of plasticity in adult visual cortex (19). The effectiveness of CVPs depends on limited plasticity in older brains; patients with early blindness are not candidates for CVPs because it is thought that in early blindness, the visual cortex is recruited for other sensory and cognitive processes, likely eliminating the retinotopic maps on which CVPs rely (20). Recruitment of the visual cortex could reflect too much plasticity at young ages. In contrast, patients with adult-onset blindness who received retinal prostheses reported little improvement on simple perceptual tasks, even after extensive training (21, 22), reflecting too little plasticity at older ages. Agents developed to aid stroke recovery (23) could find application in increasing desirable plasticity following CVP implantation.

The study by Fernández et al. brings much needed innovation to the field of CVP by showing that a single Utah array implanted in a blind human patient can reliably produce visual percepts (8). A collaborating group recently implanted a total of 16 Utah arrays in the visual cortex of a sighted monkey, which resulted in a thousand potential phosphene locations (24) and demonstrated that the approach used by Fernández et al. is scalable. However, obtaining full visual field coverage would be difficult or impossible due to the number of arrays required and the three-dimensional topography of early visual cortex in humans. More phosphenes increase the potential resolution of the presented image, but a CVP constructed with the best available neurotechnology would still have resolution orders of magnitude coarser than a computer display. One possible avenue for future progress will be replacing electrical stimulation with a more precise stimulation technology, such as optogenetics (25). Regardless of the technique used to activate neurons, our knowledge of the relationship between artificial neural activity and perception remains rudimentary and must improve before we can artificially produce a veridical representation of the visual world in the mind’s eye.

This work was supported by NIH grant R01EY023336.

Address correspondence to: Daniel Yoshor or Michael Beauchamp, Richards Medical Research Building 6A, 3700 Hamilton Walk, Philadelphia, Pennsylvania 19104, USA. Phone: 215.349.8325; Email: daniel.yoshor@pennmedicine.upenn.edu (DY); Email: michael.beauchamp@pennmedicine.upenn.edu (MSB).

Conflict of interest: MSB, WHB, and DY hold a patent related to cortical visual prostheses: US provisional patent application serial number 62/638,365, filed with the US Patent and Trademark Office on March 5, 2018, titled ‘‘Systems and Computer Implemented Methods of Conveying a Visual Image to a Blind Subject Fitted with a Visual Prosthesis.’’

Copyright: © 2021, American Society for Clinical Investigation.

Reference information: J Clin Invest. 2021;131(23):e154983. https://doi.org/10.1172/JCI154983.

See the related article at Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex.

1. Penfield W, Rasmussen T. The cerebral cortex of man: a clinical study of localization of function. JAMA. 1950;144(16):1412.
   View this article via: CrossRef Google Scholar
2. Schmidt EM, et al. Feasibility of a visual prosthesis for the blind based on intracortical microstimulation of the visual cortex. Brain. 1996;119(pt 2):507–522.
   View this article via: PubMed Google Scholar
3. Dobelle WH, Mladejovsky MG. Phosphenes produced by electrical stimulation of human occipital cortex, and their application to the development of a prosthesis for the blind. J Physiol. 1974;243(2):553–576.
   View this article via: CrossRef PubMed Google Scholar
4. Brindley GS, Lewin WS. The visual sensations produced by electrical stimulation of the medial occipital cortex. J Physiol. 1968;194(2):54–55.
   View this article via: PubMed Google Scholar
5. Bosking WH, et al. Electrical stimulation of visual cortex: relevance for the development of visual cortical prosthetics. Annu Rev Vis Sci. 2017;3:141–166.
   View this article via: CrossRef PubMed Google Scholar
6. Troyk PR. The intracortical visual prosthesis project. In: Gabel VP, ed. Artificial Vision.
7. Normann RA, et al. Toward the development of a cortically based visual neuroprosthesis. J Neural Eng. 2009;6(3):035001.
   View this article via: CrossRef PubMed Google Scholar
8. Fernández E, et al. Visual percepts evoked with an intracortical 96-channel microelectrode array inserted in human occipital cortex. J Clin Invest. 2021;131(23):e151331.
   View this article via: JCI PubMed Google Scholar
9. Normann RA, et al. A neural interface for a cortical vision prosthesis. Vision Res. 1999;39(15):2577–2587.
   View this article via: CrossRef PubMed Google Scholar
10. Vilela M, Hochberg LR. Applications of brain-computer interfaces to the control of robotic and prosthetic arms. Handb Clin Neurol. 2020;168:87–99.
    View this article via: PubMed Google Scholar
11. Andersen RA, et al. From thought to action: The brain-machine interface in posterior parietal cortex. Proc Natl Acad Sci U S A. 2019;116(52):26274–26279.
    View this article via: CrossRef PubMed Google Scholar
12. Hughes CL, et al. Neural stimulation and recording performance in human sensorimotor cortex over 1500 days. J Neural Eng. 2021;18(4):ac18ad.
    View this article via: PubMed Google Scholar
13. Winawer J, Parvizi J. Linking electrical stimulation of human primary visual cortex, size of affected cortical area, neuronal responses, and subjective experience. Neuron. 2016;92(6):1213–1219.
    View this article via: CrossRef PubMed Google Scholar
14. Murphey DK, et al. Perceiving electrical stimulation of identified human visual areas. Proc Natl Acad Sci U S A. 2009;106(13):5389–5393.
    View this article via: CrossRef PubMed Google Scholar
15. Bosking WH, et al. Saturation in phosphene size with increasing current levels delivered to human visual cortex. J Neurosci. 2017;37(30):7188–7197.
    View this article via: CrossRef PubMed Google Scholar
16. Caspi A, et al. Eye movements and the perceived location of phosphenes generated by intracranial primary visual cortex stimulation in the blind. Brain Stimul. 2021;14(4):851–860.
    View this article via: CrossRef PubMed Google Scholar
17. Oswalt D, et al. Multi-electrode stimulation evokes consistent spatial patterns of phosphenes and improves phosphene mapping in blind subjects. Brain Stimul. 2021;14(5):1356–1372.
    View this article via: CrossRef PubMed Google Scholar
18. Beauchamp MS, et al. Dynamic stimulation of visual cortex produces form vision in sighted and blind humans. Cell. 2020;181(4):774–783.
    View this article via: CrossRef PubMed Google Scholar
19. Beyeler M, et al. Learning to see again: biological constraints on cortical plasticity and the implications for sight restoration technologies. J Neural Eng. 2017;14(5):051003.
    View this article via: CrossRef PubMed Google Scholar
20. Fine I, Park JM. Blindness and human brain plasticity. Annu Rev Vis Sci. 2018;4:337–356.
    View this article via: CrossRef PubMed Google Scholar
21. Dorn JD, et al. The detection of motion by blind subjects with the epiretinal 60-electrode (Argus II) retinal prosthesis. JAMA Ophthalmol. 2013;131(2):183–189.
    View this article via: CrossRef PubMed Google Scholar
22. Castaldi E, et al. Visual BOLD response in late blind subjects with Argus II retinal prosthesis. PLoS Biol. 2016;14(10):e1002569.
    View this article via: CrossRef PubMed Google Scholar
23. Cramer SC. Drugs to enhance motor recovery after stroke. Stroke J Cereb Circ. 2015;46(10):2998–3005.
    View this article via: CrossRef PubMed Google Scholar
24. Chen X, et al. Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex. Science. 2020;370(6521):1191–1196.
    View this article via: CrossRef PubMed Google Scholar
25. Ortiz-Rios M, et al. Optogenetic stimulation of primate V1 reveals local laminar and large-scale cortical networks related to perceptual phosphenes [preprint]. https://doi.org/10.1101/2021.06.01.446505 Posted on bioRxiv June 1, 2021.