The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding

• URL: http://arxiv.org/abs/2609.10296v1
• Date: Wed, 09 Sep 2026 15:11:57 GMT
• Title: The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding
• Abstract summary: We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space.<n>Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language.<n>This semantic bottleneck enables recovery of high-level meaning without word-level alignment.
• Score: 1.6721602990895612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Non-invasive speech decoding remains constrained by the low signal-to-noise ratio of neural recordings, which makes fine-grained reconstruction of phonemes or individual words difficult. Motivated by neuroscientific evidence that high-level semantic representations are distributed across cortical regions and evolve over slower temporal scales, we hypothesize that semantic content may provide a more suitable target for non-invasive decoding than low-level acoustic or lexical features. We introduce Brain2Semantics2Text, a method that reconstructs text through an intermediate semantic embedding space. Our model maps sentence-level MEG responses into a semantic manifold and then inverts the predicted embeddings into natural language. This semantic bottleneck enables recovery of high-level meaning without word-level alignment. We describe the core principles of the approach, its implementation, and the strategies used to mitigate the challenges of learning a reliable neural-to-semantic mapping. Finally, we compare against prior non-invasive Brain2Text methods and show improved sentence-level results.

Related papers

• Margin-Regularized Structured Semantic Alignment for Brain-Language Correspondence [7.7737164302813895]
  MD-SigLIP aims to align brain embeddings with text embeddings in a shared semantic space, enabling retrieval-based decoding.<n>This formulation enables explicit modeling of the correspondence between neural representations and language semantics.<n> Experiments demonstrate state-of-the-art retrieval performance under both full-vocabulary and subset evaluation settings.
  arXiv  Detail & Related papers  (2026-08-17T14:47:15Z)
• Beyond Parallel Tracking: Interactive Multi-Feature Fusion Drives Semantic Reconstruction from Non-invasive Brain Recordings [16.302171409052637]
  Continuous semantic reconstruction from non-invasive neural recordings remains limited by the representational mismatch between semantic feature spaces and neural coding patterns.<n>Prior semantic decoders have predominantly relied on static lexical representations or dynamic contextualized representations in isolation.<n>This study introduces a multi-feature fusion framework for non-invasive semantic reconstruction.
  arXiv  Detail & Related papers  (2026-07-13T18:44:40Z)
• Cross-Subject Semantic Decoding with Shared-Space Alignment for Generalized Neural Representation Learning [80.18561652862725]
  Generalizing across subjects remains challenging in invasive neural recordings.<n>We propose a cross-subject semantic decoding framework that aligns neural responses to speech perception from multiple subjects into a shared latent space.
  arXiv  Detail & Related papers  (2026-07-03T03:59:16Z)
• MindAlign: Decoding Inner Speech from fMRI Signals via Multimodal Embedding Alignment under Limited Data [4.280521231404364]
  We propose MindAlign, a decoupled two-stage brain-to-language framework that enables open-ended text generation from fMRI signals.<n> Experiments on fMRI data collected during silent image description demonstrate that the proposed approach consistently outperforms fMRI-only and random baselines.
  arXiv  Detail & Related papers  (2026-06-15T08:30:03Z)
• SYNAPSE: Neuro-Symbolic Visual Thought-to-Text Decoding via Topological Semantic Denoising [5.012293526456192]
  We introduce SYNAPSE, a neuro-symbolic framework that stabilizes neural text generation through inference-time symbolic regularization.<n>By purifying EEG-derived semantic candidates using commonsense graph structure and latent exemplars, SYNAPSE improves semantic stability without end-to-end LLM fine-tuning.
  arXiv  Detail & Related papers  (2026-05-27T00:12:44Z)
• Brain-CLIPLM: Decoding Compressed Semantic Representations in EEG for Language Reconstruction [20.162170862752813]
  We introduce a framework that decomposes EEG-to-text decoding into semantic anchor extraction via contrastive learning and sentence reconstruction.<n>Brain-CLIPLM achieves 67.55% top-5 and 85.00% top-25 sentence retrieval accuracy, significantly outperforming direct decoding baseline.<n>Results suggest EEG-to-text decoding is better framed as recovering compressed semantic content rather than reconstructing full sentences.
  arXiv  Detail & Related papers  (2026-03-23T11:45:51Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis [55.45253486141108]
  RAG-Gesture is a diffusion-based gesture generation approach to produce semantically rich gestures.<n>We achieve this by using explicit domain knowledge to retrieve motions from a database of co-speech gestures.<n>We propose a control paradigm for guidance, that allows the users to modulate the amount of influence each retrieval insertion has over the generated sequence.
  arXiv  Detail & Related papers  (2024-12-09T18:59:46Z)
• Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition [91.39701446828144]
  We show that spiking neural networks can be trained like standard recurrent neural networks using the surrogate gradient method. They have shown promising results on speech command recognition tasks. In contrast to their recurrent non-spiking counterparts, they show robustness to exploding gradient problems without the need to use gates.
  arXiv  Detail & Related papers  (2022-12-01T12:36:26Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.