Learning to See via Epiretinal Implant Stimulation in silico with Model-Based Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2606.03118v1
• Date: Tue, 02 Jun 2026 04:03:43 GMT
• Title: Learning to See via Epiretinal Implant Stimulation in silico with Model-Based Deep Reinforcement Learning
• Abstract summary: We train a deep reinforcement learning agent that learns to assemble isotropic and anisotropic shapes to form an image.<n>We show that the agent can generate more intelligible images compared to the naive method in different virtual patients.
• Score: 3.0581668008670673
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Objective: Diseases such as age-related macular degeneration and retinitis pigmentosa cause the degradation of the photoreceptor layer. One approach to restore vision is to electrically stimulate the surviving retinal ganglion cells with a microelectrode array such as epiretinal implants. Epiretinal implants are known to generate visible anisotropic shapes elongated along the axon fascicles of neighboring retinal ganglion cells. Recent work has demonstrated that to obtain isotropic pixel-like shapes, it is possible to map axon fascicles and avoid stimulating them by inactivating electrodes or lowering stimulation current levels. Avoiding axon fascicle stimulation aims to remove brushstroke-like shapes in favor of a more reduced set of pixel-like shapes. Approach: In this study, we propose the use of isotropic and anisotropic shapes to render intelligible images on the retina of a virtual patient in a reinforcement learning environment named rlretina. The environment formalizes the task as using brushstrokes in a stroke-based rendering task. Main Results: We train a deep reinforcement learning agent that learns to assemble isotropic and anisotropic shapes to form an image. We investigate which error-based or perception-based metrics is adequate to reward the agent. The agent is trained in a model-based data generation fashion using the psychophysically validated axon map model to render images as perceived by different virtual patients. We show that the agent can generate more intelligible images compared to the naive method in different virtual patients. Significance: This work shares a new way to address epiretinal stimulation that constitutes a first step towards improving visual acuity in artificially-restored vision using anisotropic phosphenes.

Related papers

• PixCell: A generative foundation model for digital histopathology images [49.00921097924924]
  We introduce PixCell, the first diffusion-based generative foundation model for histopathology.<n>We train PixCell on PanCan-30M, a vast, diverse dataset derived from 69,184 H&E-stained whole slide images covering various cancer types.
  arXiv  Detail & Related papers  (2025-06-05T15:14:32Z)
• Intraoperative Registration by Cross-Modal Inverse Neural Rendering [61.687068931599846]
  We present a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.
  arXiv  Detail & Related papers  (2024-09-18T13:40:59Z)
• Progressive Retinal Image Registration via Global and Local Deformable Transformations [49.032894312826244]
  We propose a hybrid registration framework called HybridRetina. We use a keypoint detector and a deformation network called GAMorph to estimate the global transformation and local deformable transformation. Experiments on two widely-used datasets, FIRE and FLoRI21, show that our proposed HybridRetina significantly outperforms some state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-02T08:43:50Z)
• Optimizing Retinal Prosthetic Stimuli with Conditional Invertible Neural Networks [1.1454121287632515]
  We propose to utilize normalizing flow-based conditional invertible neural networks to optimize retinal implant stimulation in an unsupervised manner. The invertibility of these networks allows us to use them as a surrogate for the computational model of the visual system. Compared to other methods, such as trivial downsampling, linear models, and feed-forward convolutional neural networks, the flow-based invertible neural network and its conditional extension yield better visual reconstruction qualities.
  arXiv  Detail & Related papers  (2024-03-07T20:16:42Z)
• Event-Driven Imaging in Turbid Media: A Confluence of Optoelectronics and Neuromorphic Computation [9.53078750806038]
  A new optical-computational method is introduced to unveil images of targets whose visibility is severely obscured by light scattering in dense, turbid media. The scheme is human vision inspired whereby diffuse photons collected from the turbid medium are first transformed to spike trains by a dynamic vision sensor as in the retina. Image reconstruction is achieved under conditions of turbidity where an original image is unintelligible to the human eye or a digital video camera.
  arXiv  Detail & Related papers  (2023-09-13T00:38:59Z)
• Adapting Brain-Like Neural Networks for Modeling Cortical Visual Prostheses [68.96380145211093]
  Cortical prostheses are devices implanted in the visual cortex that attempt to restore lost vision by electrically stimulating neurons. Currently, the vision provided by these devices is limited, and accurately predicting the visual percepts resulting from stimulation is an open challenge. We propose to address this challenge by utilizing 'brain-like' convolutional neural networks (CNNs), which have emerged as promising models of the visual system.
  arXiv  Detail & Related papers  (2022-09-27T17:33:19Z)
• Deep Learning-Based Perceptual Stimulus Encoder for Bionic Vision [6.1739856715198]
  We propose a PSE that is trained in an end-to-end fashion to predict the electrode activation patterns required to produce a desired visual percept. We demonstrate the effectiveness of the encoder on MNIST using a psychophysically validated phosphene model tailored to individual retinal implant users.
  arXiv  Detail & Related papers  (2022-03-10T19:42:09Z)
• Sharp-GAN: Sharpness Loss Regularized GAN for Histopathology Image Synthesis [65.47507533905188]
  Conditional generative adversarial networks have been applied to generate synthetic histopathology images. We propose a sharpness loss regularized generative adversarial network to synthesize realistic histopathology images.
  arXiv  Detail & Related papers  (2021-10-27T18:54:25Z)
• Explainable Diabetic Retinopathy Detection and Retinal Image Generation [16.140110713539023]
  We propose to exploit the interpretability of deep learning application in medical diagnosis. By determining and isolating the neuron activation patterns on which diabetic retinopathy detector relies to make decisions, we demonstrate the direct relation between the isolated neuron activation and lesions for a pathological explanation. To visualize the symptom encoded in the descriptor, we propose Patho-GAN, a new network to synthesize medically plausible retinal images.
  arXiv  Detail & Related papers  (2021-07-01T08:30:04Z)
• Fooling the primate brain with minimal, targeted image manipulation [67.78919304747498]
  We propose an array of methods for creating minimal, targeted image perturbations that lead to changes in both neuronal activity and perception as reflected in behavior. Our work shares the same goal with adversarial attack, namely the manipulation of images with minimal, targeted noise that leads ANN models to misclassify the images.
  arXiv  Detail & Related papers  (2020-11-11T08:30:54Z)
• Retinopathy of Prematurity Stage Diagnosis Using Object Segmentation and Convolutional Neural Networks [68.96150598294072]
  Retinopathy of Prematurity (ROP) is an eye disorder primarily affecting premature infants with lower weights. It causes proliferation of vessels in the retina and could result in vision loss and, eventually, retinal detachment, leading to blindness. In recent years, there has been a significant effort to automate the diagnosis using deep learning. This paper builds upon the success of previous models and develops a novel architecture, which combines object segmentation and convolutional neural networks (CNN) Our proposed system first trains an object segmentation model to identify the demarcation line at a pixel level and adds the resulting mask as an additional "color" channel in
  arXiv  Detail & Related papers  (2020-04-03T14:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.