Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing

• URL: http://arxiv.org/abs/2503.10663v1
• Date: Sun, 09 Mar 2025 06:14:23 GMT
• Title: Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing
• Authors: Yang Xiao, Wang Lu, Jie Ji, Ruimeng Ye, Gen Li, Xiaolong Ma, Bo Hui,
• Abstract summary: Existing methods primarily align brain signals with real-world signals using Mean Squared Error (MSE), which solely focuses on local point-wise alignment.<n>We address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE.<n>Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11% in single-subject training and 3.81% in cross-subject training.
• Score: 21.130477722306026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with real-world signals using Mean Squared Error (MSE), which solely focuses on local point-wise alignment, and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain siginals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11\% in single-subject training and 3.81\% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available soon.

Related papers

• MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data [64.92867794764247]
  MindAligner is a framework for cross-subject brain decoding from limited fMRI data.<n>Brain Transfer Matrix (BTM) projects the brain signals of an arbitrary new subject to one of the known subjects.<n>Brain Functional Alignment module is proposed to perform soft cross-subject brain alignment under different visual stimuli.
  arXiv  Detail & Related papers  (2025-02-07T16:01:59Z)
• BrainMAP: Learning Multiple Activation Pathways in Brain Networks [77.15180533984947]
  We introduce a novel framework BrainMAP to learn Multiple Activation Pathways in Brain networks.<n>Our framework enables explanatory analyses of crucial brain regions involved in tasks.
  arXiv  Detail & Related papers  (2024-12-23T09:13:35Z)
• Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation [53.70131202548981]
  We present a two-step segmentation framework employing Knowledge-Guided Prompt Learning (KGPL) for brain MRI. Specifically, we first pre-train segmentation models on large-scale datasets with sub-optimal labels. The introduction of knowledge-wise prompts captures semantic relationships between anatomical variability and biological processes.
  arXiv  Detail & Related papers  (2024-07-31T04:32:43Z)
• Simple and Effective Transfer Learning for Neuro-Symbolic Integration [50.592338727912946]
  A potential solution to this issue is Neuro-Symbolic Integration (NeSy), where neural approaches are combined with symbolic reasoning. Most of these methods exploit a neural network to map perceptions to symbols and a logical reasoner to predict the output of the downstream task. They suffer from several issues, including slow convergence, learning difficulties with complex perception tasks, and convergence to local minima. This paper proposes a simple yet effective method to ameliorate these problems.
  arXiv  Detail & Related papers  (2024-02-21T15:51:01Z)
• Brain Decodes Deep Nets [9.302098067235507]
  We developed a tool for visualizing and analyzing large pre-trained vision models by mapping them onto the brain. Our innovation arises from a surprising usage of brain encoding: predicting brain fMRI measurements in response to images.
  arXiv  Detail & Related papers  (2023-12-03T04:36:04Z)
• MBrain: A Multi-channel Self-Supervised Learning Framework for Brain Signals [7.682832730967219]
  We study the self-supervised learning framework for brain signals that can be applied to pre-train either SEEG or EEG data. Inspired by this, we propose MBrain to learn implicit spatial and temporal correlations between different channels. Our model outperforms several state-of-the-art time series SSL and unsupervised models, and has the ability to be deployed to clinical practice.
  arXiv  Detail & Related papers  (2023-06-15T09:14:26Z)
• Connectional-Style-Guided Contextual Representation Learning for Brain Disease Diagnosis [12.172262618438173]
  We propose a connectional style contextual representation learning model (CS-CRL) to capture the intrinsic pattern of the brain. CS-CRL achieves superior accuracy in multiple brain disease diagnosis tasks across six datasets and three diseases.
  arXiv  Detail & Related papers  (2023-06-08T15:39:27Z)
• Identifying Shared Decodable Concepts in the Human Brain Using Image-Language Foundation Models [2.213723689024101]
  We introduce a method that takes advantage of high-quality pretrained multimodal representations to explore fine-grained semantic networks in the human brain. To identify such brain regions, we developed a data-driven approach to uncover visual concepts that are decodable from a massive functional magnetic resonance imaging (fMRI) dataset.
  arXiv  Detail & Related papers  (2023-06-06T03:29:47Z)
• Explainable fMRI-based Brain Decoding via Spatial Temporal-pyramid Graph Convolutional Network [0.8399688944263843]
  Existing machine learning methods for fMRI-based brain decoding either suffer from low classification performance or poor explainability. We propose a biologically inspired architecture, Spatial Temporal-pyramid Graph Convolutional Network (STpGCN), to capture the spatial-temporal graph representation of functional brain activities. We conduct extensive experiments on fMRI data under 23 cognitive tasks from Human Connectome Project (HCP) S1200.
  arXiv  Detail & Related papers  (2022-10-08T12:14:33Z)
• Deep Reinforcement Learning Guided Graph Neural Networks for Brain Network Analysis [61.53545734991802]
  We propose a novel brain network representation framework, namely BN-GNN, which searches for the optimal GNN architecture for each brain network. Our proposed BN-GNN improves the performance of traditional GNNs on different brain network analysis tasks.
  arXiv  Detail & Related papers  (2022-03-18T07:05:27Z)
• Cross-Modality Neuroimage Synthesis: A Survey [71.27193056354741]
  Multi-modality imaging improves disease diagnosis and reveals distinct deviations in tissues with anatomical properties. The existence of completely aligned and paired multi-modality neuroimaging data has proved its effectiveness in brain research. An alternative solution is to explore unsupervised or weakly supervised learning methods to synthesize the absent neuroimaging data.
  arXiv  Detail & Related papers  (2022-02-14T19:29:08Z)
• Interpretation of 3D CNNs for Brain MRI Data Classification [56.895060189929055]
  We extend the previous findings in gender differences from diffusion-tensor imaging on T1 brain MRI scans. We provide the voxel-wise 3D CNN interpretation comparing the results of three interpretation methods.
  arXiv  Detail & Related papers  (2020-06-20T17:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.