Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli

• URL: http://arxiv.org/abs/2410.04497v1
• Date: Sun, 6 Oct 2024 14:29:02 GMT
• Title: Generalizability analysis of deep learning predictions of human brain responses to augmented and semantically novel visual stimuli
• Authors: Valentyn Piskovskyi, Riccardo Chimisso, Sabrina Patania, Tom Foulsham, Giuseppe Vizzari, Dimitri Ognibene,
• Abstract summary: The purpose of this work is to investigate the soundness and utility of a neural network-based approach as a framework for exploring the impact of image enhancement techniques on visual cortex activation. We prepare a set of state-of-the-art brain encoding models, selected among the top 10 methods that participated in The Algonauts Project 2023 Challenge. We analyze their ability to make valid predictions about the effects of various image enhancement techniques on neural responses.
• Score: 0.6259089590341245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The purpose of this work is to investigate the soundness and utility of a neural network-based approach as a framework for exploring the impact of image enhancement techniques on visual cortex activation. In a preliminary study, we prepare a set of state-of-the-art brain encoding models, selected among the top 10 methods that participated in The Algonauts Project 2023 Challenge [16]. We analyze their ability to make valid predictions about the effects of various image enhancement techniques on neural responses. Given the impossibility of acquiring the actual data due to the high costs associated with brain imaging procedures, our investigation builds up on a series of experiments. Specifically, we analyze the ability of brain encoders to estimate the cerebral reaction to various augmentations by evaluating the response to augmentations targeting objects (i.e., faces and words) with known impact on specific areas. Moreover, we study the predicted activation in response to objects unseen during training, exploring the impact of semantically out-of-distribution stimuli. We provide relevant evidence for the generalization ability of the models forming the proposed framework, which appears to be promising for the identification of the optimal visual augmentation filter for a given task, model-driven design strategies as well as for AR and VR applications.

Related papers

• Towards Interpretable Visual Decoding with Attention to Brain Representations [3.254716591226115]
  Recent work has demonstrated that complex visual stimuli can be decoded from human brain activity using deep generative models.<n>We propose NeuroAdapter, a visual decoding framework that directly conditions a latent diffusion model on brain representations.<n>Our results highlight the potential of end-to-end brain-to-image decoding and establish a path toward interpreting diffusion models through the lens of visual neuroscience.
  arXiv  Detail & Related papers  (2025-09-28T01:55:55Z)
• Visualizing and Controlling Cortical Responses Using Voxel-Weighted Activation Maximization [0.0]
  Deep neural networks (DNNs) are trained on visual representations that resemble those in the human visual system.<n>We show that activation can be applied to DNN-based encoding models.<n>We generate images optimized for predicted responses in individual voxels.
  arXiv  Detail & Related papers  (2025-06-04T18:48:08Z)
• A Survey on fMRI-based Brain Decoding for Reconstructing Multimodal Stimuli [26.07986165893441]
  Decoding brain signals to reconstruct stimuli drives progress in AI, disease treatment, and brain-computer interfaces. Recent advancements in neuroimaging and image generation models have significantly improved fMRI-based decoding. This survey systematically reviews recent progress in fMRI-based brain decoding, focusing on stimulus reconstruction from passive brain signals.
  arXiv  Detail & Related papers  (2025-03-20T09:23:07Z)
• Decoding Visual Experience and Mapping Semantics through Whole-Brain Analysis Using fMRI Foundation Models [10.615012396285337]
  We develop algorithms to enhance our understanding of visual processes by incorporating whole-brain activation maps. We first compare our method with state-of-the-art approaches to decoding visual processing and show improved predictive semantic accuracy by 43%.
  arXiv  Detail & Related papers  (2024-11-11T16:51:17Z)
• MindBridge: A Cross-Subject Brain Decoding Framework [60.58552697067837]
  Brain decoding aims to reconstruct stimuli from acquired brain signals. Currently, brain decoding is confined to a per-subject-per-model paradigm. We present MindBridge, that achieves cross-subject brain decoding by employing only one model.
  arXiv  Detail & Related papers  (2024-04-11T15:46:42Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex [12.1427193917406]
  We propose an artificial neural network dubbed VISION to mimic the human brain and show how it can foster neuroscientific inquiries. VISION successfully predicts human hemodynamic responses as fMRI voxel values to visual inputs with an accuracy exceeding state-of-the-art performance by 45%.
  arXiv  Detail & Related papers  (2023-09-26T15:38:26Z)
• Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs [2.4366811507669124]
  Integration of deep learning and neuroscience has led to improvements in the analysis of brain activity. The reconstruction of visual experience from human brain activity is an area that has particularly benefited. We examine the extent to which various additional decoding techniques affect the performance of visual experience reconstruction.
  arXiv  Detail & Related papers  (2023-06-20T13:48:02Z)
• Neural Networks from Biological to Artificial and Vice Versa [6.85316573653194]
  Key contribution this paper is the investigation of the impact of a dead neuron on the performance of artificial neural networks (ANNs) The aim is to assess the potential application of the findings in the biological domain, the expected results may have significant implications for the development of effective treatment strategies for neurological disorders.
  arXiv  Detail & Related papers  (2023-06-05T17:30:07Z)
• Brain Captioning: Decoding human brain activity into images and text [1.5486926490986461]
  We present an innovative method for decoding brain activity into meaningful images and captions. Our approach takes advantage of cutting-edge image captioning models and incorporates a unique image reconstruction pipeline. We evaluate our methods using quantitative metrics for both generated captions and images.
  arXiv  Detail & Related papers  (2023-05-19T09:57:19Z)
• BI AVAN: Brain inspired Adversarial Visual Attention Network [67.05560966998559]
  We propose a brain-inspired adversarial visual attention network (BI-AVAN) to characterize human visual attention directly from functional brain activity. Our model imitates the biased competition process between attention-related/neglected objects to identify and locate the visual objects in a movie frame the human brain focuses on in an unsupervised manner.
  arXiv  Detail & Related papers  (2022-10-27T22:20:36Z)
• Backprop-Free Reinforcement Learning with Active Neural Generative Coding [84.11376568625353]
  We propose a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments. We develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference. The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.
  arXiv  Detail & Related papers  (2021-07-10T19:02:27Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor [70.2226417364135]
  It is crucial that the machine should be able to recognize the emotional state of the user with high accuracy. Deep neural networks have been used with great success in recognizing emotions. We present a new model for continuous emotion recognition based on facial expression recognition.
  arXiv  Detail & Related papers  (2020-01-31T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.