Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps

• URL: http://arxiv.org/abs/2207.03380v1
• Date: Thu, 7 Jul 2022 15:37:17 GMT
• Title: Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps
• Authors: Alexandre Pasquiou (PARIETAL, UNICOG-U992), Yair Lakretz (UNICOG-U992), John Hale, Bertrand Thirion (PARIETAL), Christophe Pallier (UNICOG-U992)
• Abstract summary: We examine the impact of test loss, training corpus and model architecture on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words. We suggest good practices for future studies aiming at explaining the human language system using neural language models.
• Score: 75.84770193489639
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks. Capitalizing on this, studies in neuroscience have started to use NLMs to study neural activity in the human brain during language processing. However, many questions remain unanswered regarding which factors determine the ability of a neural language model to capture brain activity (aka its 'brain score'). Here, we make first steps in this direction and examine the impact of test loss, training corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT), on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that (1) untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words, with the untrained LSTM outperforming the transformerbased models, being less impacted by the effect of context; (2) that training NLP models improves brain scores in the same brain regions irrespective of the model's architecture; (3) that Perplexity (test loss) is not a good predictor of brain score; (4) that training data have a strong influence on the outcome and, notably, that off-the-shelf models may lack statistical power to detect brain activations. Overall, we outline the impact of modeltraining choices, and suggest good practices for future studies aiming at explaining the human language system using neural language models.

Related papers

• Brain-like Functional Organization within Large Language Models [58.93629121400745]
  The human brain has long inspired the pursuit of artificial intelligence (AI) Recent neuroimaging studies provide compelling evidence of alignment between the computational representation of artificial neural networks (ANNs) and the neural responses of the human brain to stimuli. In this study, we bridge this gap by directly coupling sub-groups of artificial neurons with functional brain networks (FBNs) This framework links the AN sub-groups to FBNs, enabling the delineation of brain-like functional organization within large language models (LLMs)
  arXiv  Detail & Related papers  (2024-10-25T13:15:17Z)
• Large Language Model-based FMRI Encoding of Language Functions for Subjects with Neurocognitive Disorder [53.575426835313536]
  This paper explores language-related functional changes in older NCD adults using LLM-based fMRI encoding and brain scores. We analyze the correlation between brain scores and cognitive scores at both whole-brain and language-related ROI levels. Our findings reveal that higher cognitive abilities correspond to better brain scores, with correlations peaking in the middle temporal gyrus.
  arXiv  Detail & Related papers  (2024-07-15T01:09:08Z)
• Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network [16.317199232071232]
  Large Language Models (LLMs) have been shown to be effective models of the human language system. In this work, we investigate the key architectural components driving the surprising alignment of untrained models.
  arXiv  Detail & Related papers  (2024-06-21T12:54:03Z)
• fMRI predictors based on language models of increasing complexity recover brain left lateralization [4.1618731507412505]
  We show that the left-right difference in brain correlation follows a scaling law with the number of parameters. This finding reconciles computational analyses of brain activity using large language models with the classic observation from aphasic patients showing left hemisphere dominance for language.
  arXiv  Detail & Related papers  (2024-05-28T09:24:52Z)
• BrainODE: Dynamic Brain Signal Analysis via Graph-Aided Neural Ordinary Differential Equations [67.79256149583108]
  We propose a novel model called BrainODE to achieve continuous modeling of dynamic brain signals. By learning latent initial values and neural ODE functions from irregular time series, BrainODE effectively reconstructs brain signals at any time point.
  arXiv  Detail & Related papers  (2024-04-30T10:53:30Z)
• Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models [29.50162863143141]
  We compare encoding performance of various neural language models and psychologically plausible models. Surprisingly, our findings revealed that psychologically plausible models outperformed neural language models across diverse contexts.
  arXiv  Detail & Related papers  (2024-04-30T08:48:07Z)
• Neural Erosion: Emulating Controlled Neurodegeneration and Aging in AI Systems [5.720259826430462]
  We use IQ tests performed by Large Language Models (LLMs) to introduce the concept of neural erosion" This deliberate erosion involves ablating synapses or neurons, or adding Gaussian noise during or after training, resulting in a controlled progressive decline in the LLMs' performance. To the best of our knowledge, this is the first work that models neurodegeneration with text data, compared to other works that operate in the computer vision domain.
  arXiv  Detail & Related papers  (2024-03-15T18:00:00Z)
• Towards a Foundation Model for Brain Age Prediction using coVariance Neural Networks [102.75954614946258]
  Increasing brain age with respect to chronological age can reflect increased vulnerability to neurodegeneration and cognitive decline. NeuroVNN is pre-trained as a regression model on healthy population to predict chronological age. NeuroVNN adds anatomical interpretability to brain age and has a scale-free' characteristic that allows its transference to datasets curated according to any arbitrary brain atlas.
  arXiv  Detail & Related papers  (2024-02-12T14:46:31Z)
• Coupling Artificial Neurons in BERT and Biological Neurons in the Human Brain [9.916033214833407]
  This study introduces a novel, general, and effective framework to link transformer-based NLP models and neural activities in response to language. Our experimental results demonstrate 1) The activations of ANs and BNs are significantly synchronized; 2) the ANs carry meaningful linguistic/semantic information and anchor to their BN signatures; 3) the anchored BNs are interpretable in a neurolinguistic context.
  arXiv  Detail & Related papers  (2023-03-27T01:41:48Z)
• Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects [82.81964713263483]
  A popular approach to decompose the neural bases of language consists in correlating, across individuals, the brain responses to different stimuli. Here, we show that a model-based approach can reach equivalent results within subjects exposed to natural stimuli.
  arXiv  Detail & Related papers  (2021-10-12T15:30:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.