Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?

• URL: http://arxiv.org/abs/2205.01404v1
• Date: Tue, 3 May 2022 10:23:08 GMT
• Title: Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity?
• Authors: Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta and Bapi Raju Surampudi
• Abstract summary: Several popular Transformer based language models have been found to be successful for text-driven brain encoding. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks. Experiments across all 10 task representations provide the following cognitive insights.
• Score: 3.186888145772382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken stories). Encoding models based on task features are used to predict activity in different regions across the whole brain. Features from coreference resolution, NER, and shallow syntax parsing explain greater variance for the reading activity. On the other hand, for the listening activity, tasks such as paraphrase generation, summarization, and natural language inference show better encoding performance. Experiments across all 10 task representations provide the following cognitive insights: (i) language left hemisphere has higher predictive brain activity versus language right hemisphere, (ii) posterior medial cortex, temporo-parieto-occipital junction, dorsal frontal lobe have higher correlation versus early auditory and auditory association cortex, (iii) syntactic and semantic tasks display a good predictive performance across brain regions for reading and listening stimuli resp.

Related papers

• Language Reconstruction with Brain Predictive Coding from fMRI Data [28.217967547268216]
  Theory of predictive coding suggests that human brain naturally engages in continuously predicting future word representations. textscPredFT achieves current state-of-the-art decoding performance with a maximum BLEU-1 score of $27.8%$.
  arXiv  Detail & Related papers  (2024-05-19T16:06:02Z)
• Speech language models lack important brain-relevant semantics [6.626540321463248]
  Recent work has shown that text-based language models predict both text-evoked and speech-evoked brain activity to an impressive degree. This poses the question of what types of information language models truly predict in the brain.
  arXiv  Detail & Related papers  (2023-11-08T13:11:48Z)
• Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language [31.636016502455693]
  We generate supervised representations on eight Natural Language Understanding (NLU) tasks using prompt-tuning. We demonstrate that prompt-tuning yields representations that better predict neural responses to Chinese stimuli than traditional fine-tuning.
  arXiv  Detail & Related papers  (2023-10-05T06:31:01Z)
• Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations? [30.495681024162835]
  We compare prompt-tuned and fine-tuned representations in neural decoding. We find that a more brain-consistent tuning method yields representations that better correlate with brain data. This indicates that our brain encodes more fine-grained concept information than shallow syntactic information.
  arXiv  Detail & Related papers  (2023-10-03T07:34:30Z)
• Probing Brain Context-Sensitivity with Masked-Attention Generation [87.31930367845125]
  We use GPT-2 transformers to generate word embeddings that capture a fixed amount of contextual information. We then tested whether these embeddings could predict fMRI brain activity in humans listening to naturalistic text.
  arXiv  Detail & Related papers  (2023-05-23T09:36:21Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps [75.84770193489639]
  We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words. We suggest good practices for future studies aiming at explaining the human language system using neural language models.
  arXiv  Detail & Related papers  (2022-07-07T15:37:17Z)
• Toward a realistic model of speech processing in the brain with self-supervised learning [67.7130239674153]
  Self-supervised algorithms trained on the raw waveform constitute a promising candidate. We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
  arXiv  Detail & Related papers  (2022-06-03T17:01:46Z)
• Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects [82.81964713263483]
  A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
  arXiv  Detail & Related papers  (2021-10-12T15:30:21Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Brain2Word: Decoding Brain Activity for Language Generation [14.24200473508597]
  We present a model that can decode fMRI data from unseen subjects. Our model achieves 5.22% Top-1 and 13.59% Top-5 accuracy in this challenging task.
  arXiv  Detail & Related papers  (2020-09-10T10:47:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.