Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding

• URL: http://arxiv.org/abs/2603.07625v1
• Date: Sun, 08 Mar 2026 13:22:20 GMT
• Title: Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding
• Abstract summary: Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals.<n>Existing methods often suffer from degraded performance when adapting to new subjects with limited data.<n>We propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment.
• Score: 37.36610841961446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals, enabling more scalable and practical brain-computer interfaces. However, existing methods often suffer from degraded performance when adapting to new subjects with limited data, as they struggle to preserve both the semantic consistency of stimuli and the alignment of brain responses. To address these challenges, we propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment in fMRI-based cross-subject visual decoding. (1) At the stimulus level, Duala introduces a semantic alignment and relational consistency strategy that preserves intra-class similarity and inter-class separability, maintaining clear semantic boundaries during adaptation. (2) At the subject level, a distribution-based feature perturbation mechanism is developed to capture both global and subject-specific variations, enabling adaptation to individual neural representations without overfitting. Experiments on the Natural Scenes Dataset (NSD) demonstrate that Duala effectively improves alignment across subjects. Remarkably, even when fine-tuned with only about one hour of fMRI data, Duala achieves over 81.1% image-to-brain retrieval accuracy and consistently outperforms existing fine-tuning strategies in both retrieval and reconstruction. Our code is available at https://github.com/ShumengLI/Duala.

Related papers

• Coarse-to-fine Hierarchical Architecture with Sequential Mamba for Brain Reconstruction [5.732406989501624]
  We propose CHASMBrain, a novel hierarchical two-stage framework for image-to-fMRI encoding.<n>Our architecture leverages a dual-stream Mamba design to explicitly separate and process global semantic tokens and local spatial patches.<n>Our method achieves a Pearson correlation of 0.429 and an MSE of 0.261, outperforming all evaluated baselines.
  arXiv  Detail & Related papers  (2026-06-03T11:53:23Z)
• Multi-Level Bidirectional Biomimetic Learning for EEG-Based Visual Decoding [10.0396148246889]
  EEG-based visual neural decoding aims to align neural responses with visual stimuli for tasks such as image retrieval.<n>We propose MB2L, a Multi-Level Bidirectional Biomimetic Learning framework that incorporates structured physiological biases into representation learning.<n>Experiments show MB2L achieves 80.5% Top-1 and 97.6% Top-5 accuracy on zero-shot EEG-to-image retrieval.
  arXiv  Detail & Related papers  (2026-05-06T09:31:35Z)
• Bidirectional Learning of Facial Action Units and Expressions via Structured Semantic Mapping across Heterogeneous Datasets [85.74213192818668]
  We propose a Structured Semantic Mapping (SSM) framework for bidirectional AU--FE learning under different data domains and heterogeneous supervision.<n>SSM consists of three key components: (1) a shared visual backbone that learns unified facial representations from dynamic AU and FE videos; (2) semantic mediation via a Textual Semantic Prototype (TSP) module; and (3) a Dynamic Prior Mapping (DPM) module that incorporates prior knowledge derived from the Facial Action Coding System.
  arXiv  Detail & Related papers  (2026-04-12T09:08:32Z)
• Fully Differentiable Bidirectional Dual-Task Synergistic Learning for Semi-Supervised 3D Medical Image Segmentation [3.9950415168730107]
  Semi-supervised learning relaxes the need of large pixel-wise labeled datasets for image segmentation by leveraging unlabeled data.<n>We propose a fully Differentiable Bidirectional Synergistic Learning (DBiSL) framework, which seamlessly integrates and enhances four critical SSL components.
  arXiv  Detail & Related papers  (2026-02-10T03:44:24Z)
• NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment [7.860082818617649]
  Visual neural decoding seeks to reconstruct or infer perceived visual stimuli from brain activity patterns.<n>Current approaches remain constrained by the scarcity of high-quality stimulus-brain response pairs.<n>We propose a novel self-supervised architecture, named NeuroBridge, to promote effective cross-modality alignment.
  arXiv  Detail & Related papers  (2025-11-10T08:29:09Z)
• Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction [65.67001243986981]
  We propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling.<n>MindHier achieves superior semantic fidelity, 4.67x faster inference, and more deterministic results than the diffusion-based baselines.
  arXiv  Detail & Related papers  (2025-10-25T15:40:07Z)
• TS-P$^2$CL: Plug-and-Play Dual Contrastive Learning for Vision-Guided Medical Time Series Classification [26.293489194966003]
  We introduce a vision-guided paradigm that transforms 1D physiological signals into 2D pseudo-images, establishing a bridge to the visual domain.<n>We employ a dual-contrastive learning strategy, while cross-modal alignment aligns time-series dynamics with visual semantics.<n>Experiments on six MedTS datasets demonstrate that TS-P$2$CL consistently outperforms fourteen methods in both subject-dependent and subject-independent settings.
  arXiv  Detail & Related papers  (2025-09-22T13:57:58Z)
• HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentation [2.964206587462833]
  A novel semi-supervised segmentation framework, called HDC, is proposed incorporating adaptive consistency learning with a single-teacher architecture.<n>The framework introduces a hierarchical distillation mechanism with two objectives: Correlation Guidance Loss for aligning feature representations and Mutual Information Loss for stabilizing noisy student learning.
  arXiv  Detail & Related papers  (2025-04-14T04:52:24Z)
• Two in One Go: Single-stage Emotion Recognition with Decoupled Subject-context Transformer [78.35816158511523]
  We present a single-stage emotion recognition approach, employing a Decoupled Subject-Context Transformer (DSCT) for simultaneous subject localization and emotion classification. We evaluate our single-stage framework on two widely used context-aware emotion recognition datasets, CAER-S and EMOTIC.
  arXiv  Detail & Related papers  (2024-04-26T07:30:32Z)
• MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion [7.597218661195779]
  We propose a two-stage image reconstruction model called MindDiffuser. In Stage 1, the VQ-VAE latent representations and the CLIP text embeddings decoded from fMRI are put into Stable Diffusion. In Stage 2, we utilize the CLIP visual feature decoded from fMRI as supervisory information, and continually adjust the two feature vectors decoded in Stage 1 through backpagation to align the structural information.
  arXiv  Detail & Related papers  (2023-08-08T13:28:34Z)
• Joint fMRI Decoding and Encoding with Latent Embedding Alignment [77.66508125297754]
  We introduce a unified framework that addresses both fMRI decoding and encoding. Our model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework.
  arXiv  Detail & Related papers  (2023-03-26T14:14:58Z)
• Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation [73.84166499988443]
  We present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA) Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. Experimental results on two different tasks demonstrate that our SIFA method is effective in improving segmentation performance on unlabeled target images.
  arXiv  Detail & Related papers  (2020-02-06T13:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.