Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence

• URL: http://arxiv.org/abs/2608.16975v1
• Date: Mon, 17 Aug 2026 14:47:15 GMT
• Title: Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence
• Abstract summary: MD-SigLIP aims to align brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding.<n>This formulation enables explicit modeling of the correspondence between neural representations and language semantics.<n> Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.
• Score: 7.7737164302813895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of large language models, brain-language decoding has achieved remarkable progress. However, it remains unclear whether decoded content genuinely reflects neural representations or is largely reconstructed by the language model itself. This ambiguity limits interpretability and hinders the investigation of intrinsic brain-language correspondence. To address this challenge, we propose MD-SigLIP. This margin-regularized structured semantic alignment framework directly aligns brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding. This formulation enables explicit modeling of the correspondence between neural representations and language semantics. Building upon duplicate-aware sigmoid contrastive learning, we introduce a listwise margin-regularized term that enforces structured ranking constraints between positive semantic clusters and negative samples. By modeling multi-positive semantic structure and margin-based ordering simultaneously, the method captures the manifold organization of language embeddings reflected in neural signals. Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.

Related papers

• The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding [1.6721602990895612]
  We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space.<n>Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language.<n>This semantic bottleneck enables recovery of high-level meaning without word-level alignment.
  arXiv  Detail & Related papers  (2026-09-09T15:11:57Z)
• Cross-Subject Semantic Decoding with Shared-Space Alignment for Generalized Neural Representation Learning [80.18561652862725]
  Generalizing across subjects remains challenging in invasive neural recordings.<n>We propose a cross-subject semantic decoding framework that aligns neural responses to speech perception from multiple subjects into a shared latent space.
  arXiv  Detail & Related papers  (2026-07-03T03:59:16Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data [4.280521231404364]
  We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals.<n> Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines.
  arXiv  Detail & Related papers  (2026-06-15T08:30:03Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• Pixel Sentence Representation Learning [67.4775296225521]
  In this work, we conceptualize the learning of sentence-level textual semantics as a visual representation learning process. We employ visually-grounded text perturbation methods like typos and word order shuffling, resonating with human cognitive patterns, and enabling perturbation to be perceived as continuous. Our approach is further bolstered by large-scale unsupervised topical alignment training and natural language inference supervision.
  arXiv  Detail & Related papers  (2024-02-13T02:46:45Z)
• Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling [0.0]
  The black-box structure of the deep neural network in pre-trained language models seriously limits the interpretability of the language modeling process. A Word-Context-Coupled Space (W2CSpace) is proposed by introducing the alignment processing between uninterpretable neural representation and interpretable statistical logic. Our language model can achieve better performance and highly credible interpretable ability compared to related state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-19T09:26:02Z)
• The Whole Truth and Nothing But the Truth: Faithful and Controllable Dialogue Response Generation with Dataflow Transduction and Constrained Decoding [65.34601470417967]
  We describe a hybrid architecture for dialogue response generation that combines the strengths of neural language modeling and rule-based generation. Our experiments show that this system outperforms both rule-based and learned approaches in human evaluations of fluency, relevance, and truthfulness.
  arXiv  Detail & Related papers  (2022-09-16T09:00:49Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Does injecting linguistic structure into language models lead to better alignment with brain recordings? [13.880819301385854]
  We evaluate whether language models align better with brain recordings if their attention is biased by annotations from syntactic or semantic formalisms. Our proposed approach enables the evaluation of more targeted hypotheses about the composition of meaning in the brain.
  arXiv  Detail & Related papers  (2021-01-29T14:42:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.