Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?

• URL: http://arxiv.org/abs/2310.01854v1
• Date: Tue, 3 Oct 2023 07:34:30 GMT
• Title: Fine-tuned vs. Prompt-tuned Supervised Representations: Which Better Account for Brain Language Representations?
• Authors: Jingyuan Sun and Marie-Francine Moens
• Abstract summary: We compare prompt-tuned and fine-tuned representations in neural decoding. We find that a more brain-consistent tuning method yields representations that better correlate with brain data. This indicates that our brain encodes more fine-grained concept information than shallow syntactic information.
• Score: 30.495681024162835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To decipher the algorithm underlying the human brain's language representation, previous work probed brain responses to language input with pre-trained artificial neural network (ANN) models fine-tuned on NLU tasks. However, full fine-tuning generally updates the entire parametric space and distorts pre-trained features, cognitively inconsistent with the brain's robust multi-task learning ability. Prompt-tuning, in contrast, protects pre-trained weights and learns task-specific embeddings to fit a task. Could prompt-tuning generate representations that better account for the brain's language representations than fine-tuning? If so, what kind of NLU task leads a pre-trained model to better decode the information represented in the human brain? We investigate these questions by comparing prompt-tuned and fine-tuned representations in neural decoding, that is predicting the linguistic stimulus from the brain activities evoked by the stimulus. We find that on none of the 10 NLU tasks, full fine-tuning significantly outperforms prompt-tuning in neural decoding, implicating that a more brain-consistent tuning method yields representations that better correlate with brain data. Moreover, we identify that tasks dealing with fine-grained concept meaning yield representations that better decode brain activation patterns than other tasks, especially the syntactic chunking task. This indicates that our brain encodes more fine-grained concept information than shallow syntactic information when representing languages.

Related papers

• Improving semantic understanding in speech language models via brain-tuning [19.732593005537606]
  Speech language models align with human brain responses to natural language to an impressive degree. Current models rely heavily on low-level speech features, indicating they lack brain-relevant semantics. We address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with fMRI recordings.
  arXiv  Detail & Related papers  (2024-10-11T20:06:21Z)
• Investigating the Encoding of Words in BERT's Neurons using Feature Textualization [11.943486282441143]
  We propose a technique to produce representations of neurons in embedding word space. We find that the produced representations can provide insights about the encoded knowledge in individual neurons.
  arXiv  Detail & Related papers  (2023-11-14T15:21:49Z)
• Tuning In to Neural Encoding: Linking Human Brain and Artificial Supervised Representations of Language [31.636016502455693]
  We generate supervised representations on eight Natural Language Understanding (NLU) tasks using prompt-tuning. We demonstrate that prompt-tuning yields representations that better predict neural responses to Chinese stimuli than traditional fine-tuning.
  arXiv  Detail & Related papers  (2023-10-05T06:31:01Z)
• Probing Brain Context-Sensitivity with Masked-Attention Generation [87.31930367845125]
  We use GPT-2 transformers to generate word embeddings that capture a fixed amount of contextual information. We then tested whether these embeddings could predict fMRI brain activity in humans listening to naturalistic text.
  arXiv  Detail & Related papers  (2023-05-23T09:36:21Z)
• BrainBERT: Self-supervised representation learning for intracranial recordings [18.52962864519609]
  We create a reusable Transformer, BrainBERT, for intracranial recordings bringing modern representation learning approaches to neuroscience. Much like in NLP and speech recognition, this Transformer enables classifying complex concepts, with higher accuracy and with much less data. In the future, far more concepts will be decodable from neural recordings by using representation learning, potentially unlocking the brain like language models unlocked language.
  arXiv  Detail & Related papers  (2023-02-28T07:40:37Z)
• Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps [75.84770193489639]
  We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words. We suggest good practices for future studies aiming at explaining the human language system using neural language models.
  arXiv  Detail & Related papers  (2022-07-07T15:37:17Z)
• Toward a realistic model of speech processing in the brain with self-supervised learning [67.7130239674153]
  Self-supervised algorithms trained on the raw waveform constitute a promising candidate. We show that Wav2Vec 2.0 learns brain-like representations with as little as 600 hours of unlabelled speech.
  arXiv  Detail & Related papers  (2022-06-03T17:01:46Z)
• Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? [3.186888145772382]
  Several popular Transformer based language models have been found to be successful for text-driven brain encoding. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks. Experiments across all 10 task representations provide the following cognitive insights.
  arXiv  Detail & Related papers  (2022-05-03T10:23:08Z)
• Emergence of Machine Language: Towards Symbolic Intelligence with Neural Networks [73.94290462239061]
  We propose to combine symbolism and connectionism principles by using neural networks to derive a discrete representation. By designing an interactive environment and task, we demonstrated that machines could generate a spontaneous, flexible, and semantic language.
  arXiv  Detail & Related papers  (2022-01-14T14:54:58Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Compositional Explanations of Neurons [52.71742655312625]
  We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts. We use this procedure to answer several questions on interpretability in models for vision and natural language processing.
  arXiv  Detail & Related papers  (2020-06-24T20:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.