Bridging Auditory Perception and Language Comprehension through MEG-Driven Encoding Models

• URL: http://arxiv.org/abs/2501.03246v1
• Date: Sun, 22 Dec 2024 19:41:54 GMT
• Title: Bridging Auditory Perception and Language Comprehension through MEG-Driven Encoding Models
• Authors: Matteo Ciferri, Matteo Ferrante, Nicola Toschi,
• Abstract summary: We use Magnetoencephalography (MEG) data to analyze brain responses to spoken language stimuli. We develop two distinct encoding models: an audio-to-MEG encoder, and a text-to-MEG encoder. Both models successfully predict neural activity, demonstrating significant correlations between estimated and observed MEG signals.
• Score: 0.12289361708127873
• License:
• Abstract: Understanding the neural mechanisms behind auditory and linguistic processing is key to advancing cognitive neuroscience. In this study, we use Magnetoencephalography (MEG) data to analyze brain responses to spoken language stimuli. We develop two distinct encoding models: an audio-to-MEG encoder, which uses time-frequency decompositions (TFD) and wav2vec2 latent space representations, and a text-to-MEG encoder, which leverages CLIP and GPT-2 embeddings. Both models successfully predict neural activity, demonstrating significant correlations between estimated and observed MEG signals. However, the text-to-MEG model outperforms the audio-based model, achieving higher Pearson Correlation (PC) score. Spatially, we identify that auditory-based embeddings (TFD and wav2vec2) predominantly activate lateral temporal regions, which are responsible for primary auditory processing and the integration of auditory signals. In contrast, textual embeddings (CLIP and GPT-2) primarily engage the frontal cortex, particularly Broca's area, which is associated with higher-order language processing, including semantic integration and language production, especially in the 8-30 Hz frequency range. The strong involvement of these regions suggests that auditory stimuli are processed through more direct sensory pathways, while linguistic information is encoded via networks that integrate meaning and cognitive control. Our results reveal distinct neural pathways for auditory and linguistic information processing, with higher encoding accuracy for text representations in the frontal regions. These insights refine our understanding of the brain's functional architecture in processing auditory and textual information, offering quantitative advancements in the modelling of neural responses to complex language stimuli.

Related papers

• BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation [29.78480739360263]
  We propose a new multi-stage strategy for semantic brain signal decoding via vEctor-quantized speCtrogram reconstruction. BrainECHO successively conducts: 1) autoencoding of the audio spectrogram; 2) Brain-audio latent space alignment; and 3) Semantic text generation via Whisper finetuning. BrainECHO outperforms state-of-the-art methods under the same data split settings on two widely accepted resources.
  arXiv  Detail & Related papers  (2024-10-19T04:29:03Z)
• Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder [53.575426835313536]
  This paper explores language-related functional changes in older NCD adults using LLM-based fMRI encoding and brain scores. We analyze the correlation between brain scores and cognitive scores at both whole-brain and language-related ROI levels. Our findings reveal that higher cognitive abilities correspond to better brain scores, with correlations peaking in the middle temporal gyrus.
  arXiv  Detail & Related papers  (2024-07-15T01:09:08Z)
• Du-IN: Discrete units-guided mask modeling for decoding speech from Intracranial Neural signals [5.283718601431859]
  Invasive brain-computer interfaces with Electrocorticography (ECoG) have shown promise for high-performance speech decoding in medical applications. We developed the Du-IN model, which extracts contextual embeddings based on region-level tokens through discrete codex-guided mask modeling. Our model achieves state-of-the-art performance on the 61-word classification task, surpassing all baselines.
  arXiv  Detail & Related papers  (2024-05-19T06:00:36Z)
• Do self-supervised speech and language models extract similar representations as human brain? [2.390915090736061]
  Speech and language models trained through self-supervised learning (SSL) demonstrate strong alignment with brain activity during speech and language perception. We evaluate the brain prediction performance of two representative SSL models, Wav2Vec2.0 and GPT-2.
  arXiv  Detail & Related papers  (2023-10-07T01:39:56Z)
• Probing Brain Context-Sensitivity with Masked-Attention Generation [87.31930367845125]
  We use GPT-2 transformers to generate word embeddings that capture a fixed amount of contextual information. We then tested whether these embeddings could predict fMRI brain activity in humans listening to naturalistic text.
  arXiv  Detail & Related papers  (2023-05-23T09:36:21Z)
• Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context [87.31930367845125]
  We trained a lexical language model, Glove, and a supra-lexical language model, GPT-2, on a text corpus. We then assessed to what extent these information-restricted models were able to predict the time-courses of fMRI signal of humans listening to naturalistic text. Our analyses show that, while most brain regions involved in language are sensitive to both syntactic and semantic variables, the relative magnitudes of these effects vary a lot across these regions.
  arXiv  Detail & Related papers  (2023-02-28T08:16:18Z)
• Toward a realistic model of speech processing in the brain with self-supervised learning [67.7130239674153]
  Self-supervised algorithms trained on the raw waveform constitute a promising candidate. We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
  arXiv  Detail & Related papers  (2022-06-03T17:01:46Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• CogAlign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals [60.921888445317705]
  We propose a CogAlign approach to integrate cognitive language processing signals into natural language processing models. We show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets.
  arXiv  Detail & Related papers  (2021-06-10T07:10:25Z)
• Extracting the Locus of Attention at a Cocktail Party from Single-Trial EEG using a Joint CNN-LSTM Model [0.1529342790344802]
  Human brain performs remarkably well in segregating a particular speaker from interfering speakers in a multi-speaker scenario. We present a joint convolutional neural network (CNN) - long short-term memory (LSTM) model to infer the auditory attention.
  arXiv  Detail & Related papers  (2021-02-08T01:06:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.