Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings

• URL: http://arxiv.org/abs/2607.12071v2
• Date: Mon, 20 Jul 2026 01:49:37 GMT
• Title: Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings
• Abstract summary: Continuous semantic reconstruction from non-invasive neural recordings remains limited by the representational mismatch between semantic feature spaces and neural coding patterns.<n>Prior semantic decoders have predominantly relied on static lexical representations or dynamic contextualized representations in isolation.<n>This study introduces a multi-feature fusion framework for non-invasive semantic reconstruction.
• Score: 16.302171409052637
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous semantic reconstruction from non-invasive neural recordings remains limited by the representational mismatch between semantic feature spaces and neural coding patterns, which severely impedes cross-modal alignment between high-noise neural signals and target semantic features. Prior semantic decoders have predominantly relied on static lexical representations or dynamic contextualized representations in isolation. This single-dimension approach inevitably leads to severe information loss, as it fails to account for the human brain's capacity to integrate stable word attributes and dynamic contexts simultaneously. To bridge this gap, this study introduces a multi-feature fusion framework for non-invasive semantic reconstruction, systematically benchmarking two integration approaches: linear Naive Concatenation and non-linear Multi-Head Cross-Attention. Within this framework, our approach complements static lexical representations (W2V) with dynamic contextual representations (GPT) via an interactive gating mechanism to facilitate cooperative processing during language comprehension. Evaluated through extensive semantic reconstruction and text generation experiments, our framework reveals a robust performance hierarchy: Cross-Att > Concat > GPT > W2V. Crucially, the non-linear cross-attention fusion method achieves state-of-the-art performance, demonstrating that neural language decoding benefits from simulating the collaborative modulation between contextual information and core lexical attributes rather than depending on isolated individual features, while also offering a viable non-invasive brain-to-text decoding method.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence [7.7737164302813895]
  MD-SigLIP aims to align brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding.<n>This formulation enables explicit modeling of the correspondence between neural representations and language semantics.<n> Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.
  arXiv  Detail & Related papers  (2026-08-17T14:47:15Z)
• Cross-Subject Semantic Decoding with Shared-Space Alignment for Generalized Neural Representation Learning [80.18561652862725]
  Generalizing across subjects remains challenging in invasive neural recordings.<n>We propose a cross-subject semantic decoding framework that aligns neural responses to speech perception from multiple subjects into a shared latent space.
  arXiv  Detail & Related papers  (2026-07-03T03:59:16Z)
• SABER: A Semantic-Aligned Brain Network Analysis Framework via Multi-scale Hypergraphs [2.525239819116781]
  Brain disease diagnosis requires synergy of brain connectivity patterns and high-level semantic knowledge.<n>Existing methods treat semantics from large language models (LLMs) as auxiliary features or supervision.<n>We propose a semantic-aligned brain network framework that actively integrates LLM-derived semantics into the prediction process.
  arXiv  Detail & Related papers  (2026-07-02T08:58:47Z)
• Vision-Core Guided Contrastive Learning for Balanced Multi-modal Prognosis Prediction of Stroke [4.272151451487239]
  This paper proposes a novel tri-modal fusion model for ischemic stroke prognosis.<n>We employ a Large Language Model (LLM) to automatically generate semi-structured diagnostic text from brain MRIs.<n>We also design a core component termed the Vision-Conditioned Dual Alignment Fusion Module.
  arXiv  Detail & Related papers  (2026-05-14T11:28:57Z)
• Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition [37.30058027224552]
  Multimodal emotion recognition in conversations aims to infer utterance-level emotions by jointly modeling textual, acoustic, and visual cues within context.<n>We propose a framework that combines dual-space feature disentanglement with dual-branch graph learning.<n> Experiments on IEMOCAP and MELD demonstrate that the proposed method achieves superior performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-03T14:47:26Z)
• Contrastive and Multi-Task Learning on Noisy Brain Signals with Nonlinear Dynamical Signatures [5.37454752035459]
  We introduce a two-stage multitask learning framework for analyzing EEG signals.<n>In the first stage, a denoising autoencoder is trained to suppress artifacts and stabilize temporal dynamics.<n>In the second stage, a multitask architecture processes these denoised signals to achieve three objectives.
  arXiv  Detail & Related papers  (2026-01-13T13:36:38Z)
• Boosting Neural Language Inference via Cascaded Interactive Reasoning [38.125341836302525]
  Natural Language Inference (NLI) focuses on ascertaining the logical relationship between a given premise and hypothesis.<n>This task presents significant challenges due to inherent linguistic features such as diverse phrasing, semantic complexity, and contextual nuances.<n>We introduce the Cascaded Interactive Reasoning Network (CIRN), a novel architecture designed for deeper semantic comprehension in NLI.
  arXiv  Detail & Related papers  (2025-05-10T11:37:15Z)
• Hierarchical Banzhaf Interaction for General Video-Language Representation Learning [60.44337740854767]
  Multimodal representation learning plays an important role in the artificial intelligence domain.<n>We introduce a new approach that models video-text as game players using multivariate cooperative game theory.<n>We extend our original structure into a flexible encoder-decoder framework, enabling the model to adapt to various downstream tasks.
  arXiv  Detail & Related papers  (2024-12-30T14:09:15Z)
• Neuron: Learning Context-Aware Evolving Representations for Zero-Shot Skeleton Action Recognition [64.56321246196859]
  We propose a novel dyNamically Evolving dUal skeleton-semantic syneRgistic framework.<n>We first construct the spatial-temporal evolving micro-prototypes and integrate dynamic context-aware side information.<n>We introduce the spatial compression and temporal memory mechanisms to guide the growth of spatial-temporal micro-prototypes.
  arXiv  Detail & Related papers  (2024-11-18T05:16:11Z)
• Spatial Semantic Recurrent Mining for Referring Image Segmentation [63.34997546393106]
  We propose Stextsuperscript2RM to achieve high-quality cross-modality fusion. It follows a working strategy of trilogy: distributing language feature, spatial semantic recurrent coparsing, and parsed-semantic balancing. Our proposed method performs favorably against other state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-05-15T00:17:48Z)
• VIRT: Improving Representation-based Models for Text Matching through Virtual Interaction [50.986371459817256]
  We propose a novel textitVirtual InteRacTion mechanism, termed as VIRT, to enable full and deep interaction modeling in representation-based models. VIRT asks representation-based encoders to conduct virtual interactions to mimic the behaviors as interaction-based models do.
  arXiv  Detail & Related papers  (2021-12-08T09:49:28Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.